{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "name": "12_Recurrent_Neural_Networks",
      "provenance": [],
      "collapsed_sections": [],
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eTdCMVl9YAXw",
        "colab_type": "text"
      },
      "source": [
        "<a href=\"https://practicalai.me\"><img src=\"https://raw.githubusercontent.com/practicalAI/images/master/images/rounded_logo.png\" width=\"100\" align=\"left\" hspace=\"20px\" vspace=\"20px\"></a>\n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/practicalAI/images/master/images/12_Recurrent_Neural_Networks/simple_rnn.png\" width=\"250\" align=\"right\">\n",
        "\n",
        "<div align=\"left\">\n",
        "<h1>Recurrent Neural Networks (RNN) </h1>\n",
        "\n",
        "In this lesson we will learn how to process sequential data (sentences, time-series, etc.) with recurrent neural networks (RNNs). "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "xuabAj4PYj57",
        "colab_type": "text"
      },
      "source": [
        "<table align=\"center\">\n",
        "  <td>\n",
        "<img src=\"https://raw.githubusercontent.com/practicalAI/images/master/images/rounded_logo.png\" width=\"25\"><a target=\"_blank\" href=\"https://practicalai.me\"> View on practicalAI</a>\n",
        "  </td>\n",
        "  <td>\n",
        "<img src=\"https://raw.githubusercontent.com/practicalAI/images/master/images/colab_logo.png\" width=\"25\"><a target=\"_blank\" href=\"https://colab.research.google.com/github/practicalAI/practicalAI/blob/master/notebooks/12_Recurrent_Neural_Networks.ipynb\"> Run in Google Colab</a>\n",
        "  </td>\n",
        "  <td>\n",
        "<img src=\"https://raw.githubusercontent.com/practicalAI/images/master/images/github_logo.png\" width=\"22\"><a target=\"_blank\" href=\"https://github.com/practicalAI/practicalAI/blob/master/notebooks/12_Recurrent_Neural_Networks.ipynb\"> View code on GitHub</a>\n",
        "  </td>\n",
        "</table>"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "uXuhm14WaCxy",
        "colab_type": "text"
      },
      "source": [
        "So far we've processed inputs as whole (ex. applying filters across the entire input to extract features) but we can also processed our inputs sequentially. To illustrate what this looks like, suppose our inputs are sentences (like the news dataset inputs we've seen in previous lessons). Think of each word/puncutation (or token) in the sentence as a timestep. So a sentence with 8 tokens has 8 timesteps. We can process each timestep, one at a time, and predict the class after the last timestep (token) has been processed. This is very powerful because the model now has a meaningful way to account for the order in our sequence and predict accordingly. "
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "m5JJY0tl8StF",
        "colab_type": "text"
      },
      "source": [
        "# Overview"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JqxyljU18hvt",
        "colab_type": "text"
      },
      "source": [
        "* **Objective:**  Process sequential data by accounting for the currend input and also what has been learned from previous inputs.\n",
        "* **Advantages:** \n",
        "    * Account for order and previous inputs in a meaningful way.\n",
        "    * Conditioned generation for generating sequences.\n",
        "* **Disadvantages:** \n",
        "    * Each time step's prediction depends on the previous prediction so it's difficult to parallelize RNN operations. \n",
        "    * Processing long sequences can yield memory and computation issues.\n",
        "    * Interpretability is difficult but there are few [techniques](https://arxiv.org/abs/1506.02078) that use the activations from RNNs to see what parts of the inputs are processed. \n",
        "* **Miscellaneous:** \n",
        "    * Architectural tweaks to make RNNs faster and interpretable is an ongoing area of research."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "oSOqIKzckhfc",
        "colab_type": "text"
      },
      "source": [
        "# Set up"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "vSPqM7rrkhE8",
        "colab_type": "code",
        "outputId": "38ef452c-e050-4d8a-d31b-e14b43d292bd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Use TensorFlow 2.x\n",
        "%tensorflow_version 2.x"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "TensorFlow 2.x selected.\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GuK4QHvikkpR",
        "colab_type": "code",
        "outputId": "128c7a2a-daa7-487e-ba04-2b7f957a0081",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "import os\n",
        "import numpy as np\n",
        "import tensorflow as tf\n",
        "print(\"GPU Available: \", tf.test.is_gpu_available())"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "GPU Available:  True\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0Zwz1m7mknyd",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Arguments\n",
        "SEED = 1234\n",
        "SHUFFLE = True\n",
        "DATA_FILE = 'news.csv'\n",
        "INPUT_FEATURE = 'title'\n",
        "OUTPUT_FEATURE = 'category'\n",
        "FILTERS = \"!\\\"'#$%&()*+,-./:;<=>?@[\\\\]^_`{|}~\"\n",
        "LOWER = True\n",
        "CHAR_LEVEL = False\n",
        "TRAIN_SIZE = 0.7\n",
        "VAL_SIZE = 0.15\n",
        "TEST_SIZE = 0.15\n",
        "NUM_EPOCHS = 10\n",
        "BATCH_SIZE = 256\n",
        "EMBEDDING_DIM = 100\n",
        "RNN_HIDDEN_DIM = 128\n",
        "RNN_DROPOUT_P = 0.1\n",
        "NUM_LAYERS = 1\n",
        "HIDDEN_DIM = 100\n",
        "DROPOUT_P = 0.1\n",
        "LEARNING_RATE = 1e-3\n",
        "EARLY_STOPPING_CRITERIA = 3"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "HgEiBe6DkpXV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Set seed for reproducability\n",
        "np.random.seed(SEED)\n",
        "tf.random.set_seed(SEED)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "c69z9wpJ56nE",
        "colab_type": "text"
      },
      "source": [
        "# Data"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2V_nEp5G58M0",
        "colab_type": "text"
      },
      "source": [
        "We will download the [AG News dataset](http://www.di.unipi.it/~gulli/AG_corpus_of_news_articles.html), which consists of 120000 text samples from 4 unique classes ('Business', 'Sci/Tech', 'Sports', 'World')"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "y3qKSoEe57na",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import pandas as pd\n",
        "import re\n",
        "import urllib"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "cGQo98566GIV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Upload data from GitHub to notebook's local drive\n",
        "url = \"https://raw.githubusercontent.com/practicalAI/practicalAI/master/data/news.csv\"\n",
        "response = urllib.request.urlopen(url)\n",
        "html = response.read()\n",
        "with open(DATA_FILE, 'wb') as fp:\n",
        "    fp.write(html)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dG_Oltib6G-9",
        "colab_type": "code",
        "outputId": "eebcc768-99c4-4932-b3a2-bab6ea733920",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 204
        }
      },
      "source": [
        "# Load data\n",
        "df = pd.read_csv(DATA_FILE, header=0)\n",
        "X = df[INPUT_FEATURE].values\n",
        "y = df[OUTPUT_FEATURE].values\n",
        "df.head(5)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/html": [
              "<div>\n",
              "<style scoped>\n",
              "    .dataframe tbody tr th:only-of-type {\n",
              "        vertical-align: middle;\n",
              "    }\n",
              "\n",
              "    .dataframe tbody tr th {\n",
              "        vertical-align: top;\n",
              "    }\n",
              "\n",
              "    .dataframe thead th {\n",
              "        text-align: right;\n",
              "    }\n",
              "</style>\n",
              "<table border=\"1\" class=\"dataframe\">\n",
              "  <thead>\n",
              "    <tr style=\"text-align: right;\">\n",
              "      <th></th>\n",
              "      <th>title</th>\n",
              "      <th>category</th>\n",
              "    </tr>\n",
              "  </thead>\n",
              "  <tbody>\n",
              "    <tr>\n",
              "      <th>0</th>\n",
              "      <td>Wall St. Bears Claw Back Into the Black (Reuters)</td>\n",
              "      <td>Business</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>1</th>\n",
              "      <td>Carlyle Looks Toward Commercial Aerospace (Reu...</td>\n",
              "      <td>Business</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>2</th>\n",
              "      <td>Oil and Economy Cloud Stocks' Outlook (Reuters)</td>\n",
              "      <td>Business</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>3</th>\n",
              "      <td>Iraq Halts Oil Exports from Main Southern Pipe...</td>\n",
              "      <td>Business</td>\n",
              "    </tr>\n",
              "    <tr>\n",
              "      <th>4</th>\n",
              "      <td>Oil prices soar to all-time record, posing new...</td>\n",
              "      <td>Business</td>\n",
              "    </tr>\n",
              "  </tbody>\n",
              "</table>\n",
              "</div>"
            ],
            "text/plain": [
              "                                               title  category\n",
              "0  Wall St. Bears Claw Back Into the Black (Reuters)  Business\n",
              "1  Carlyle Looks Toward Commercial Aerospace (Reu...  Business\n",
              "2    Oil and Economy Cloud Stocks' Outlook (Reuters)  Business\n",
              "3  Iraq Halts Oil Exports from Main Southern Pipe...  Business\n",
              "4  Oil prices soar to all-time record, posing new...  Business"
            ]
          },
          "metadata": {
            "tags": []
          },
          "execution_count": 44
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "hxo6RKCQ71dl",
        "colab_type": "text"
      },
      "source": [
        "# Split data"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "eS6kCcfY6IHE",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import collections\n",
        "from sklearn.model_selection import train_test_split"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Mt3yeNsYXU_F",
        "colab_type": "text"
      },
      "source": [
        "### Components"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0gbh2TLBXVDu",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def train_val_test_split(X, y, val_size, test_size, shuffle):\n",
        "    \"\"\"Split data into train/val/test datasets.\n",
        "    \"\"\"\n",
        "    X_train, X_test, y_train, y_test = train_test_split(\n",
        "        X, y, test_size=test_size, stratify=y, shuffle=shuffle)\n",
        "    X_train, X_val, y_train, y_val = train_test_split(\n",
        "        X_train, y_train, test_size=val_size, stratify=y_train, shuffle=shuffle)\n",
        "    return X_train, X_val, X_test, y_train, y_val, y_test"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "8XIdYU_n7536",
        "colab_type": "text"
      },
      "source": [
        "### Operations"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "kqiQd2j_76gP",
        "colab_type": "code",
        "outputId": "7f1d72dd-11f1-4d31-cebd-9d00caf91524",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 119
        }
      },
      "source": [
        "# Create data splits\n",
        "X_train, X_val, X_test, y_train, y_val, y_test = train_val_test_split(\n",
        "    X=X, y=y, val_size=VAL_SIZE, test_size=TEST_SIZE, shuffle=SHUFFLE)\n",
        "class_counts = dict(collections.Counter(y))\n",
        "print (f\"X_train: {X_train.shape}, y_train: {y_train.shape}\")\n",
        "print (f\"X_val: {X_val.shape}, y_val: {y_val.shape}\")\n",
        "print (f\"X_test: {X_test.shape}, y_test: {y_test.shape}\")\n",
        "print (f\"X_train[0]: {X_train[0]}\")\n",
        "print (f\"y_train[0]: {y_train[0]}\")\n",
        "print (f\"Classes: {class_counts}\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "X_train: (86700,), y_train: (86700,)\n",
            "X_val: (15300,), y_val: (15300,)\n",
            "X_test: (18000,), y_test: (18000,)\n",
            "X_train[0]: With protesters waiting, Powell cancels trip to Olympics\n",
            "y_train[0]: Sports\n",
            "Classes: {'Business': 30000, 'Sci/Tech': 30000, 'Sports': 30000, 'World': 30000}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "dIfmW7vJ8Jx1",
        "colab_type": "text"
      },
      "source": [
        "# Tokenizer"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "JP4VCO0LAJUt",
        "colab_type": "text"
      },
      "source": [
        "Unlike the previous notebook, we will be processing our text at a word-level (as opposed to character-level)."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "DHPAxkKR7736",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.preprocessing.text import Tokenizer\n",
        "from tensorflow.keras.utils import to_categorical"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EKAXyjYHfEzU",
        "colab_type": "text"
      },
      "source": [
        "### Components"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bVgBSnFTfE5D",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def untokenize(indices, tokenizer):\n",
        "    \"\"\"Untokenize a list of indices into string.\"\"\"\n",
        "    return \" \".join([tokenizer.index_word[index] for index in indices])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "_BD3XPKF8L84",
        "colab_type": "text"
      },
      "source": [
        "### Operations"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WcscM_vL8KvP",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Input vectorizer\n",
        "X_tokenizer = Tokenizer(filters=FILTERS,\n",
        "                        lower=LOWER,\n",
        "                        char_level=CHAR_LEVEL,\n",
        "                        oov_token='<UNK>')"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "xV2JgpOA8PwO",
        "colab_type": "code",
        "outputId": "b116f5dc-692f-4dfb-8d97-8dde0ceeb69d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Fit only on train data\n",
        "X_tokenizer.fit_on_texts(X_train)\n",
        "vocab_size = len(X_tokenizer.word_index) + 1\n",
        "print (f\"# tokens: {vocab_size}\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "# tokens: 29872\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ybb-YZSz8Qno",
        "colab_type": "code",
        "outputId": "742e501f-043e-4cb6-fac0-a8a8ea59cf2b",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "# Convert text to sequence of tokens\n",
        "original_text = X_train[0]\n",
        "X_train = np.array(X_tokenizer.texts_to_sequences(X_train))\n",
        "X_val = np.array(X_tokenizer.texts_to_sequences(X_val))\n",
        "X_test = np.array(X_tokenizer.texts_to_sequences(X_test))\n",
        "preprocessed_text = untokenize(X_train[0], X_tokenizer)\n",
        "print (f\"{original_text} \\n\\t→ {preprocessed_text} \\n\\t→ {X_train[0]}\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "SCO caps legal costs as losses mount \n",
            "\t→ sco caps legal costs as losses mount \n",
            "\t→ [2451, 3953, 762, 568, 21, 1015, 1249]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ORGuhjCf8TKh",
        "colab_type": "text"
      },
      "source": [
        "# LabelEncoder"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7aBBgzkW8Rxv",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from sklearn.preprocessing import LabelEncoder"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "z_jVCsl98U09",
        "colab_type": "text"
      },
      "source": [
        "### Operations"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "ckM_MnQi8UTH",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Output vectorizer\n",
        "y_tokenizer = LabelEncoder()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0-FkxqCT8WUk",
        "colab_type": "code",
        "outputId": "5d5dc244-95df-4bc6-b93f-5bbe3cf0be99",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Fit on train data\n",
        "y_tokenizer = y_tokenizer.fit(y_train)\n",
        "classes = list(y_tokenizer.classes_)\n",
        "print (f\"classes: {classes}\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "classes: ['Business', 'Sci/Tech', 'Sports', 'World']\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yrLHd1i_8XAJ",
        "colab_type": "code",
        "outputId": "913fa40c-307e-4e98-d065-226a65311914",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Convert labels to tokens\n",
        "y_train = y_tokenizer.transform(y_train)\n",
        "y_val = y_tokenizer.transform(y_val)\n",
        "y_test = y_tokenizer.transform(y_test)\n",
        "print (f\"y_train[0]: {y_train[0]}\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "y_train[0]: 2\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bJAR-FDUSP4q",
        "colab_type": "code",
        "outputId": "d8a30a83-5474-4e03-dc8b-cab3eb706dd3",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "# Class weights\n",
        "counts = np.bincount(y_train)\n",
        "class_weights = {i: 1.0/count for i, count in enumerate(counts)}\n",
        "print (f\"class counts: {counts},\\nclass weights: {class_weights}\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "class counts: [21675 21675 21675 21675],\n",
            "class weights: {0: 4.61361014994233e-05, 1: 4.61361014994233e-05, 2: 4.61361014994233e-05, 3: 4.61361014994233e-05}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "eoWQk0hO9bK2",
        "colab_type": "text"
      },
      "source": [
        "# Generators"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "GVxnbzgW8X1V",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import math\n",
        "from tensorflow.keras.preprocessing.sequence import pad_sequences\n",
        "from tensorflow.keras.utils import Sequence"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IMtHyqex9gVI",
        "colab_type": "text"
      },
      "source": [
        "### Components"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1w6wVKJe9fxk",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class DataGenerator(Sequence):\n",
        "    \"\"\"Custom data loader.\"\"\"\n",
        "    def __init__(self, X, y, batch_size, shuffle=True):\n",
        "        self.X = X\n",
        "        self.y = y\n",
        "        self.batch_size = batch_size\n",
        "        self.shuffle = shuffle\n",
        "        self.on_epoch_end()\n",
        "\n",
        "    def __len__(self):\n",
        "        \"\"\"# of batches.\"\"\"\n",
        "        return math.ceil(len(self.X) / self.batch_size)\n",
        "\n",
        "    def __str__(self):\n",
        "        return (f\"<DataGenerator(\" \\\n",
        "                f\"batch_size={self.batch_size}, \" \\\n",
        "                f\"batches={len(self)}, \" \\\n",
        "                f\"shuffle={self.shuffle})>\")\n",
        "\n",
        "    def __getitem__(self, index):\n",
        "        \"\"\"Generate a batch.\"\"\"\n",
        "\n",
        "        # Gather indices for this batch\n",
        "        batch_indices = self.epoch_indices[index*self.batch_size:(index+1)*self.batch_size]\n",
        "\n",
        "        # Generate batch data\n",
        "        inputs, outputs = self.create_batch(batch_indices=batch_indices)\n",
        "\n",
        "        return inputs, outputs\n",
        "\n",
        "    def on_epoch_end(self):\n",
        "        \"\"\"Create indices after each epoch.\"\"\"\n",
        "        self.epoch_indices = np.arange(len(self.X))\n",
        "        if self.shuffle == True:\n",
        "            np.random.shuffle(self.epoch_indices)\n",
        "\n",
        "    def create_batch(self, batch_indices):\n",
        "        \"\"\"Generate data from indices.\"\"\"\n",
        "        X = self.X[batch_indices]\n",
        "        y = self.y[batch_indices]\n",
        "        \n",
        "        # Sequence lengths\n",
        "        seq_lengths = np.array([[i, len(x)-1] for i, x in enumerate(X)])\n",
        "\n",
        "        # Pad batch\n",
        "        max_seq_len = max([len(x) for x in X])\n",
        "        X = pad_sequences(X, padding=\"post\", maxlen=max_seq_len)\n",
        "\n",
        "        return [X, seq_lengths], y"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "u37JyFYV9ilS",
        "colab_type": "text"
      },
      "source": [
        "### Operations"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "5T8mVj9d9hNI",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Dataset generator\n",
        "training_generator = DataGenerator(X=X_train,\n",
        "                                   y=y_train,\n",
        "                                   batch_size=BATCH_SIZE,\n",
        "                                   shuffle=SHUFFLE)\n",
        "validation_generator = DataGenerator(X=X_val,\n",
        "                                     y=y_val,\n",
        "                                     batch_size=BATCH_SIZE,\n",
        "                                     shuffle=False)\n",
        "testing_generator = DataGenerator(X=X_test,\n",
        "                                  y=y_test,\n",
        "                                  batch_size=BATCH_SIZE,\n",
        "                                  shuffle=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "drbY5WDX9kcL",
        "colab_type": "code",
        "outputId": "79462796-a13d-447f-9435-f67b6b83716d",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "print (f\"training_generator: {training_generator}\")\n",
        "print (f\"validation_generator: {validation_generator}\")\n",
        "print (f\"testing_generator: {testing_generator}\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "training_generator: <DataGenerator(batch_size=256, batches=339, shuffle=True)>\n",
            "validation_generator: <DataGenerator(batch_size=256, batches=60, shuffle=False)>\n",
            "testing_generator: <DataGenerator(batch_size=256, batches=71, shuffle=False)>\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GN_ezwx7vBUo",
        "colab_type": "text"
      },
      "source": [
        "# Input"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "EwyHfs4JvC7F",
        "colab_type": "text"
      },
      "source": [
        "Inputs to RNNs are sequential like text or time-series."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "pSCOyGL_vPu5",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.layers import Input"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "JOs64ORxvBei",
        "colab_type": "code",
        "outputId": "22d20104-82af-4435-8c30-d93197c5eb5a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Input\n",
        "sequence_size = 8 # words per input\n",
        "x = Input(shape=(sequence_size, EMBEDDING_DIM))\n",
        "print (x)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Tensor(\"input_1:0\", shape=(None, 8, 100), dtype=float32)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "bUMibnFtuq_i",
        "colab_type": "text"
      },
      "source": [
        "# Simple RNN"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Ufxzi5G0DScl",
        "colab_type": "text"
      },
      "source": [
        "<img src=\"https://raw.githubusercontent.com/practicalAI/images/master/images/12_Recurrent_Neural_Networks/simple_rnn.png\" width=\"500\">\n",
        "\n",
        "RNN forward pass for a single time step $X_t$:\n",
        "\n",
        "$h_t = tanh(W_{hh}h_{t-1} + W_{xh}X_t+b_h)$\n",
        "\n",
        "*where*:\n",
        "* $W_{hh}$ = hidden units weights| $\\in \\mathbb{R}^{HXH}$ ($H$ is the hidden dim)\n",
        "* $h_{t-1}$ = previous timestep's hidden state $\\in \\mathbb{R}^{NXH}$\n",
        "* $W_{xh}$ = input weights| $\\in \\mathbb{R}^{EXH}$\n",
        "* $X_t$ = input at time step t | $\\in \\mathbb{R}^{NXE}$ ($N$ is the batch size, $E$ is the embedding dim)\n",
        "* $b_h$ = hidden units bias $\\in \\mathbb{R}^{HX1}$\n",
        "* $h_t$ = output from RNN for timestep $t$\n",
        "\n",
        "<div align=\"left\">\n",
        "<img src=\"https://raw.githubusercontent.com/practicalAI/images/master/images/lightbulb.gif\" width=\"45px\" align=\"left\" hspace=\"10px\" vspace=\"10px\">\n",
        "</div>\n",
        "\n",
        "At the first time step, the previous hidden state $h_{t-1}$ can either be a zero vector (unconditioned) or initialized (conditioned). If we are conditioning the RNN, the first hidden state $h_0$ can belong to a specific condition or we can concat the specific condition to the randomly initialized hidden vectors at each time step. More on this in the subsequent notebooks on RNNs."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "_jJ5TmJ2ut3z",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.layers import SimpleRNN"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z-O2Du78ut8J",
        "colab_type": "code",
        "outputId": "5f9fbd89-294f-4644-f0c1-09ea6d128b2a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "# RNN forward pass (many to one)\n",
        "rnn = SimpleRNN(units=RNN_HIDDEN_DIM, \n",
        "                dropout=DROPOUT_P, \n",
        "                recurrent_dropout=RNN_DROPOUT_P,\n",
        "                return_sequences=False, # only get the output from the last sequential input\n",
        "                return_state=True)\n",
        "output, hidden_state = rnn(x)\n",
        "print (f\"output {output.shape}\")\n",
        "print (f\"hidden {hidden_state.shape}\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "output (None, 128)\n",
            "hidden (None, 128)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FqOtTdDx3Nz7",
        "colab_type": "code",
        "outputId": "7e99fcaf-aa6e-4911-8a04-44baa8950c22",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 51
        }
      },
      "source": [
        "# RNN forward pass (many to many)\n",
        "rnn = SimpleRNN(units=RNN_HIDDEN_DIM, \n",
        "                dropout=DROPOUT_P, \n",
        "                recurrent_dropout=RNN_DROPOUT_P,\n",
        "                return_sequences=True, # get outputs from every item in sequential input\n",
        "                return_state=True)\n",
        "outputs, hidden_state = rnn(x)\n",
        "print (f\"output {outputs.shape}\")\n",
        "print (f\"hidden {hidden_state.shape}\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "output (None, 8, 128)\n",
            "hidden (None, 128)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "mnUh3iRpJ2j8",
        "colab_type": "text"
      },
      "source": [
        "There are many different ways to use RNNs. So far we've processed our inputs one timestep at a time and we could either use the RNN's output at each time step or just use the final input timestep's RNN output. Let's look at a few other possibilities.\n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/practicalAI/images/master/images/12_Recurrent_Neural_Networks/rnn_examples.png\" width=\"1000\">"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "pfhjWZRD94hK",
        "colab_type": "text"
      },
      "source": [
        "# Model"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "zVmJGm8m-KIz",
        "colab_type": "text"
      },
      "source": [
        "**Simple RNN**"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "T8oCutDJ-d1J",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import tensorflow.keras.backend as K\n",
        "from tensorflow.keras.layers import Dense\n",
        "from tensorflow.keras.layers import Dropout\n",
        "from tensorflow.keras.layers import Embedding\n",
        "from tensorflow.keras.layers import Input\n",
        "from tensorflow.keras.layers import Masking\n",
        "from tensorflow.keras.losses import SparseCategoricalCrossentropy\n",
        "from tensorflow.keras.models import Model\n",
        "from tensorflow.keras.optimizers import Adam"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6B2MewCdCeKC",
        "colab_type": "text"
      },
      "source": [
        "### Components"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UPP5ROd69mXC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class TextClassificationRNNModel(Model):\n",
        "    def __init__(self, vocab_size, embedding_dim, rnn_cell, \n",
        "                 hidden_dim, dropout_p, num_classes):\n",
        "        super(TextClassificationRNNModel, self).__init__()\n",
        "\n",
        "        # Embeddings\n",
        "        self.embedding = Embedding(input_dim=vocab_size,\n",
        "                                   output_dim=embedding_dim,\n",
        "                                   mask_zero=True,\n",
        "                                   trainable=True)\n",
        "        \n",
        "        # Masking\n",
        "        self.mask = Masking(mask_value=0.)\n",
        "        \n",
        "        # RNN\n",
        "        self.rnn = rnn_cell\n",
        "        \n",
        "        # FC layers\n",
        "        self.fc1 = Dense(units=hidden_dim, activation='relu')\n",
        "        self.dropout = Dropout(rate=dropout_p)\n",
        "        self.fc2 = Dense(units=num_classes, activation='softmax')\n",
        "\n",
        "    def call(self, inputs, training=False):\n",
        "        \"\"\"Forward pass.\"\"\"\n",
        "        \n",
        "        # Inputs\n",
        "        x_in, seq_lengths = inputs\n",
        "\n",
        "        # Embed\n",
        "        x_emb = self.embedding(x_in)\n",
        "        \n",
        "        # Masking\n",
        "        z = self.mask(x_emb)\n",
        "\n",
        "        # RNN\n",
        "        z, hidden_state = self.rnn(x_emb)\n",
        "\n",
        "        # Gather last relevant index\n",
        "        z = tf.gather_nd(z, K.cast(seq_lengths, 'int32'))\n",
        "\n",
        "        # FC\n",
        "        z = self.fc1(z)\n",
        "        if training:\n",
        "            z = self.dropout(z, training=training)\n",
        "        y_pred = self.fc2(z)\n",
        "\n",
        "        return y_pred\n",
        "    \n",
        "    def sample(self, x_in_shape, seq_lengths_shape):\n",
        "        x_in = Input(shape=x_in_shape)\n",
        "        seq_lengths = Input(shape=seq_lengths_shape)\n",
        "        inputs = [x_in, seq_lengths]\n",
        "        return Model(inputs=inputs, outputs=self.call(inputs)).summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "ADmeEdSZj56Y",
        "colab_type": "text"
      },
      "source": [
        "In our model, we want to use the RNN's output after the last relevant token in the sentence is processed. The last relevant token doesn't refer the <PAD> tokens but to the last actual word in the sentence and its index is different for each input in the batch.\n",
        "\n",
        "This is why we included a `seq_lengths` variable in our `create_batch` function in the `Generator` class. The seq_lengths are passed in to the forward pass as an input and we use [tf.gather_nd](https://www.tensorflow.org/api_docs/python/tf/gather_nd) to gather the last relevant hidden state (before padding starts). \n",
        "\n",
        "```\n",
        "def create_batch(self, batch_indices):\n",
        "    ...\n",
        "    # Sequence lengths\n",
        "    seq_lengths = np.array([[i, len(x)-1] for i, x in enumerate(X)])\n",
        "    ...\n",
        "    return [X, seq_lengths], y\n",
        "```\n",
        "\n",
        "Once we have the relevant hidden state extracted, we proceed to apply some fully-connected layers (with softmax) to generate the class probabilities."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "H39ggqTI5XfP",
        "colab_type": "text"
      },
      "source": [
        "# Inspection"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "GJfY2QZbz2WL",
        "colab_type": "text"
      },
      "source": [
        "Let's say you want more transparency on the exact shapes of the inputs and outputs at every stage of your model. Well you could architect a model that just works and you can use model.sample() to see the summary but what if you have issues in the model in the first place? "
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1Rei-P5DRYFC",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Get the first data point\n",
        "sample_inputs, sample_y = training_generator.create_batch([0]) "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "aJOozQkzK5jA",
        "colab_type": "code",
        "outputId": "3c8f8342-8234-4752-bc6e-4c477477673a",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "sample_X, sample_seq_length = sample_inputs\n",
        "print (f\"sample_X: {sample_X} ==> shape: {sample_X.shape}\")\n",
        "print (f\"sample_seq_length: {sample_seq_length} ==> shape: {sample_seq_length.shape}\")\n",
        "print (f\"sample_y: {sample_y} ==> shape: {sample_y.shape}\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "sample_X: [[  18 2087 2186  709 1653 1202    2  672]] ==> shape: (1, 8)\n",
            "sample_seq_length: [[0 7]] ==> shape: (1, 2)\n",
            "sample_y: [2] ==> shape: (1,)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "0_j0XG3h3DzZ",
        "colab_type": "text"
      },
      "source": [
        "Now we will create a smaller model to the point that we want to inspect. This is a great way to iteratively build your model, validating the output shapes at every step of the way."
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "2H6nmXq82GJo",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class ModelToInspect(Model):\n",
        "    def __init__(self, vocab_size, embedding_dim, rnn_cell, \n",
        "                 hidden_dim, dropout_p, num_classes):\n",
        "        super(ModelToInspect, self).__init__()\n",
        "\n",
        "        # Embeddings\n",
        "        self.embedding = Embedding(input_dim=vocab_size,\n",
        "                                   output_dim=embedding_dim,\n",
        "                                   mask_zero=True,\n",
        "                                   trainable=True)\n",
        "        \n",
        "        # Masking\n",
        "        self.mask = Masking(mask_value=0.)\n",
        "        \n",
        "        # RNN\n",
        "        self.rnn = rnn_cell\n",
        "\n",
        "    def call(self, inputs, training=False):\n",
        "        \"\"\"Forward pass.\"\"\"\n",
        "        \n",
        "        # Forward pass\n",
        "        x_in, seq_lengths = inputs\n",
        "        x_emb = self.embedding(x_in)\n",
        "        z = self.mask(x_emb)\n",
        "        z, hidden_state = self.rnn(x_emb)\n",
        "        \n",
        "        return z\n",
        "    \n",
        "    def sample(self, x_in_shape, seq_lengths_shape):\n",
        "        x_in = Input(shape=x_in_shape)\n",
        "        seq_lengths = Input(shape=seq_lengths_shape)\n",
        "        inputs = [x_in, seq_lengths]\n",
        "        return Model(inputs=inputs, outputs=self.call(inputs)).summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fkGjkJW02a2a",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# RNN cell\n",
        "simple_rnn = SimpleRNN(units=RNN_HIDDEN_DIM, \n",
        "                       dropout=DROPOUT_P, \n",
        "                       recurrent_dropout=RNN_DROPOUT_P,\n",
        "                       return_sequences=True,\n",
        "                       return_state=True) "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "dEOvGxPC2azp",
        "colab_type": "code",
        "outputId": "f07fcb89-5c6c-4083-bde8-8e63f46cdbcb",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 289
        }
      },
      "source": [
        "model = ModelToInspect(vocab_size=vocab_size,\n",
        "                       embedding_dim=EMBEDDING_DIM,\n",
        "                       rnn_cell=simple_rnn,\n",
        "                       hidden_dim=HIDDEN_DIM,\n",
        "                       dropout_p=DROPOUT_P,\n",
        "                       num_classes=len(classes))\n",
        "model.sample(x_in_shape=(sample_X.shape[1],), seq_lengths_shape=(2,))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input_2 (InputLayer)            [(None, 8)]          0                                            \n",
            "__________________________________________________________________________________________________\n",
            "embedding (Embedding)           (None, 8, 100)       2987200     input_2[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "input_3 (InputLayer)            [(None, 2)]          0                                            \n",
            "__________________________________________________________________________________________________\n",
            "simple_rnn_3 (SimpleRNN)        [(None, 8, 128), (No 29312       embedding[0][0]                  \n",
            "==================================================================================================\n",
            "Total params: 3,016,512\n",
            "Trainable params: 3,016,512\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NJNTVi0dR8s-",
        "colab_type": "code",
        "outputId": "4eab615e-30b1-4cd9-85de-a5cbd27f1aa9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 238
        }
      },
      "source": [
        "z = model(sample_inputs)\n",
        "print (f\"z: {z} ==> shape: {z.shape}\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "z: [[[ 0.00820867 -0.02330007 -0.02288972 ... -0.03135864  0.00753112\n",
            "   -0.01934274]\n",
            "  [ 0.04829089  0.01333961  0.04919123 ... -0.11085248 -0.01596617\n",
            "    0.02034008]\n",
            "  [ 0.06126435 -0.02994554  0.00135956 ... -0.03555164  0.10374444\n",
            "   -0.03651815]\n",
            "  ...\n",
            "  [-0.0012784   0.10571703  0.04443682 ...  0.0003274  -0.00568285\n",
            "   -0.04744783]\n",
            "  [-0.04378117 -0.13406402  0.13039635 ...  0.14210068  0.08378843\n",
            "   -0.10440754]\n",
            "  [-0.01132762 -0.08137172 -0.1193947  ...  0.10263453 -0.01641675\n",
            "    0.02320403]]] ==> shape: (1, 8, 128)\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "QtqZKYks21ga",
        "colab_type": "text"
      },
      "source": [
        "The mode.sample() provided a decent summary of the shapes but with our close inspection strategy, we are able to see actual values for a specific input."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "S9ggYO6yHIm2",
        "colab_type": "text"
      },
      "source": [
        "# Training"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "geKOPVzVK6S9",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.callbacks import Callback\n",
        "from tensorflow.keras.callbacks import EarlyStopping\n",
        "from tensorflow.keras.callbacks import ReduceLROnPlateau\n",
        "from tensorflow.keras.callbacks import TensorBoard\n",
        "%load_ext tensorboard"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "zsWl3U9UdiJ6",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# RNN cell\n",
        "simple_rnn = SimpleRNN(units=RNN_HIDDEN_DIM, \n",
        "                       dropout=DROPOUT_P, \n",
        "                       recurrent_dropout=RNN_DROPOUT_P,\n",
        "                       return_sequences=True,\n",
        "                       return_state=True) "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "wD4sRUS5_lwq",
        "colab_type": "code",
        "outputId": "7d2f8ab5-1b30-484b-821d-6cc516eaf857",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 442
        }
      },
      "source": [
        "model = TextClassificationRNNModel(vocab_size=vocab_size,\n",
        "                                   embedding_dim=EMBEDDING_DIM,\n",
        "                                   rnn_cell=simple_rnn,\n",
        "                                   hidden_dim=HIDDEN_DIM,\n",
        "                                   dropout_p=DROPOUT_P,\n",
        "                                   num_classes=len(classes))\n",
        "model.sample(x_in_shape=(sequence_size,), seq_lengths_shape=(2,))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_1\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input_4 (InputLayer)            [(None, 8)]          0                                            \n",
            "__________________________________________________________________________________________________\n",
            "embedding_1 (Embedding)         (None, 8, 100)       2987200     input_4[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "input_5 (InputLayer)            [(None, 2)]          0                                            \n",
            "__________________________________________________________________________________________________\n",
            "simple_rnn_4 (SimpleRNN)        [(None, 8, 128), (No 29312       embedding_1[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "tf_op_layer_Cast (TensorFlowOpL [(None, 2)]          0           input_5[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "tf_op_layer_GatherNd (TensorFlo [(None, 128)]        0           simple_rnn_4[0][0]               \n",
            "                                                                 tf_op_layer_Cast[0][0]           \n",
            "__________________________________________________________________________________________________\n",
            "dense (Dense)                   (None, 100)          12900       tf_op_layer_GatherNd[0][0]       \n",
            "__________________________________________________________________________________________________\n",
            "dense_1 (Dense)                 (None, 4)            404         dense[0][0]                      \n",
            "==================================================================================================\n",
            "Total params: 3,029,816\n",
            "Trainable params: 3,029,816\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Ucn3tYq1_sE1",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Compile\n",
        "model.compile(optimizer=Adam(lr=LEARNING_RATE),\n",
        "              loss=SparseCategoricalCrossentropy(),\n",
        "              metrics=['accuracy'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "X5ac-uJXb-F7",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Callbacks\n",
        "callbacks = [EarlyStopping(monitor='val_loss', patience=EARLY_STOPPING_CRITERIA, verbose=1, mode='min'),\n",
        "             ReduceLROnPlateau(patience=1, factor=0.1, verbose=0),\n",
        "             TensorBoard(log_dir='tensorboard/simple_rnn', histogram_freq=1, update_freq='epoch')]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qlqe2TVlCxvj",
        "colab_type": "code",
        "outputId": "12c59594-0934-4293-d953-3d7e545efe20",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 374
        }
      },
      "source": [
        "# Training\n",
        "training_history = model.fit_generator(generator=training_generator,\n",
        "                                       epochs=NUM_EPOCHS,\n",
        "                                       validation_data=validation_generator,\n",
        "                                       callbacks=callbacks,\n",
        "                                       shuffle=False,\n",
        "                                       class_weight=class_weights,\n",
        "                                       verbose=1)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "339/339 [==============================] - 46s 136ms/step - loss: 5.8538e-05 - accuracy: 0.2998 - val_loss: 0.8294 - val_accuracy: 0.6992\n",
            "Epoch 2/10\n",
            "339/339 [==============================] - 45s 132ms/step - loss: 2.9162e-05 - accuracy: 0.7392 - val_loss: 0.4876 - val_accuracy: 0.8310\n",
            "Epoch 3/10\n",
            "339/339 [==============================] - 45s 132ms/step - loss: 1.9858e-05 - accuracy: 0.8528 - val_loss: 0.4259 - val_accuracy: 0.8525\n",
            "Epoch 4/10\n",
            "339/339 [==============================] - 44s 130ms/step - loss: 1.6225e-05 - accuracy: 0.8854 - val_loss: 0.4136 - val_accuracy: 0.8569\n",
            "Epoch 5/10\n",
            "339/339 [==============================] - 43s 128ms/step - loss: 1.4041e-05 - accuracy: 0.9010 - val_loss: 0.4134 - val_accuracy: 0.8607\n",
            "Epoch 6/10\n",
            "339/339 [==============================] - 43s 128ms/step - loss: 1.2500e-05 - accuracy: 0.9146 - val_loss: 0.4392 - val_accuracy: 0.8529\n",
            "Epoch 7/10\n",
            "339/339 [==============================] - 43s 127ms/step - loss: 9.8448e-06 - accuracy: 0.9303 - val_loss: 0.4103 - val_accuracy: 0.8632\n",
            "Epoch 8/10\n",
            "339/339 [==============================] - 43s 125ms/step - loss: 9.3332e-06 - accuracy: 0.9347 - val_loss: 0.4139 - val_accuracy: 0.8628\n",
            "Epoch 9/10\n",
            "339/339 [==============================] - 42s 125ms/step - loss: 8.8823e-06 - accuracy: 0.9387 - val_loss: 0.4146 - val_accuracy: 0.8631\n",
            "Epoch 10/10\n",
            "339/339 [==============================] - 43s 127ms/step - loss: 8.8024e-06 - accuracy: 0.9381 - val_loss: 0.4147 - val_accuracy: 0.8632\n",
            "Epoch 00010: early stopping\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "fylHduxTK-0N",
        "colab_type": "code",
        "outputId": "656639a9-1e4e-46e3-dc85-e9112ba4f690",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Evaluation\n",
        "testing_history = model.evaluate_generator(generator=testing_generator, \n",
        "                                           verbose=1)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "71/71 [==============================] - 2s 35ms/step - loss: 0.3991 - accuracy: 0.8673\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "gaJD2YGpcTfT",
        "colab_type": "code",
        "outputId": "ed0f00a4-2164-460d-94f0-57a0778953f2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 17
        }
      },
      "source": [
        "%tensorboard --logdir tensorboard"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "    <div id=\"root\"></div>\n",
              "    <script>\n",
              "      (function() {\n",
              "        window.TENSORBOARD_ENV = window.TENSORBOARD_ENV || {};\n",
              "        window.TENSORBOARD_ENV[\"IN_COLAB\"] = true;\n",
              "        document.querySelector(\"base\").href = \"https://localhost:6006\";\n",
              "        function fixUpTensorboard(root) {\n",
              "          const tftb = root.querySelector(\"tf-tensorboard\");\n",
              "          // Disable the fragment manipulation behavior in Colab. Not\n",
              "          // only is the behavior not useful (as the iframe's location\n",
              "          // is not visible to the user), it causes TensorBoard's usage\n",
              "          // of `window.replace` to navigate away from the page and to\n",
              "          // the `localhost:<port>` URL specified by the base URI, which\n",
              "          // in turn causes the frame to (likely) crash.\n",
              "          tftb.removeAttribute(\"use-hash\");\n",
              "        }\n",
              "        function executeAllScripts(root) {\n",
              "          // When `script` elements are inserted into the DOM by\n",
              "          // assigning to an element's `innerHTML`, the scripts are not\n",
              "          // executed. Thus, we manually re-insert these scripts so that\n",
              "          // TensorBoard can initialize itself.\n",
              "          for (const script of root.querySelectorAll(\"script\")) {\n",
              "            const newScript = document.createElement(\"script\");\n",
              "            newScript.type = script.type;\n",
              "            newScript.textContent = script.textContent;\n",
              "            root.appendChild(newScript);\n",
              "            script.remove();\n",
              "          }\n",
              "        }\n",
              "        function setHeight(root, height) {\n",
              "          // We set the height dynamically after the TensorBoard UI has\n",
              "          // been initialized. This avoids an intermediate state in\n",
              "          // which the container plus the UI become taller than the\n",
              "          // final width and cause the Colab output frame to be\n",
              "          // permanently resized, eventually leading to an empty\n",
              "          // vertical gap below the TensorBoard UI. It's not clear\n",
              "          // exactly what causes this problematic intermediate state,\n",
              "          // but setting the height late seems to fix it.\n",
              "          root.style.height = `${height}px`;\n",
              "        }\n",
              "        const root = document.getElementById(\"root\");\n",
              "        fetch(\".\")\n",
              "          .then((x) => x.text())\n",
              "          .then((html) => void (root.innerHTML = html))\n",
              "          .then(() => fixUpTensorboard(root))\n",
              "          .then(() => executeAllScripts(root))\n",
              "          .then(() => setHeight(root, 800));\n",
              "      })();\n",
              "    </script>\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "lFWSgL8E5_HU",
        "colab_type": "text"
      },
      "source": [
        "# Bidirectional RNN"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "sO_vP00z6BiJ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.layers import Bidirectional"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "WqtGrVhm7914",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "class TextClassificationBiRNNModel(Model):\n",
        "    def __init__(self, vocab_size, embedding_dim, rnn_cell, \n",
        "                 hidden_dim, dropout_p, num_classes):\n",
        "        super(TextClassificationBiRNNModel, self).__init__()\n",
        "\n",
        "        # Embeddings\n",
        "        self.embedding = Embedding(input_dim=vocab_size,\n",
        "                                   output_dim=embedding_dim,\n",
        "                                   mask_zero=True,\n",
        "                                   trainable=True)\n",
        "        \n",
        "        # Masking\n",
        "        self.mask = Masking(mask_value=0.)\n",
        "        \n",
        "        # RNN\n",
        "        self.rnn = Bidirectional(rnn_cell, merge_mode='sum')\n",
        "        \n",
        "        # FC layers\n",
        "        self.fc1 = Dense(units=hidden_dim, activation='relu')\n",
        "        self.dropout = Dropout(rate=dropout_p)\n",
        "        self.fc2 = Dense(units=num_classes, activation='softmax')\n",
        "\n",
        "    def call(self, inputs, training=False):\n",
        "        \"\"\"Forward pass.\"\"\"\n",
        "        \n",
        "        # Inputs\n",
        "        x_in, seq_lengths = inputs\n",
        "\n",
        "        # Embed\n",
        "        x_emb = self.embedding(x_in)\n",
        "        \n",
        "        # Masking\n",
        "        z = self.mask(x_emb)\n",
        "\n",
        "        # RNN\n",
        "        z, hidden_state_fw, hidden_state_bw = self.rnn(x_emb)\n",
        "\n",
        "        # Gather last relevant index\n",
        "        z = tf.gather_nd(z, K.cast(seq_lengths, 'int32'))\n",
        "\n",
        "        # FC\n",
        "        z = self.fc1(z)\n",
        "        if training:\n",
        "            z = self.dropout(z, training=training)\n",
        "        y_pred = self.fc2(z)\n",
        "\n",
        "        return y_pred\n",
        "    \n",
        "    def sample(self, x_in_shape, seq_lengths_shape):\n",
        "        x_in = Input(shape=x_in_shape)\n",
        "        seq_lengths = Input(shape=seq_lengths_shape)\n",
        "        inputs = [x_in, seq_lengths]\n",
        "        return Model(inputs=inputs, outputs=self.call(inputs)).summary()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "yzz9ctXv6BmZ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Bidirectional RNN cell\n",
        "simple_rnn = SimpleRNN(units=RNN_HIDDEN_DIM, \n",
        "                       dropout=DROPOUT_P, \n",
        "                       recurrent_dropout=RNN_DROPOUT_P,\n",
        "                       return_sequences=True,\n",
        "                       return_state=True)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "FkqYRBWd6OCR",
        "colab_type": "code",
        "outputId": "14e5bdd0-b1fe-4906-842e-2c7c6ad055b8",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 442
        }
      },
      "source": [
        "model = TextClassificationBiRNNModel(vocab_size=vocab_size,\n",
        "                                     embedding_dim=EMBEDDING_DIM,\n",
        "                                     rnn_cell=simple_rnn,\n",
        "                                     hidden_dim=HIDDEN_DIM,\n",
        "                                     dropout_p=DROPOUT_P,\n",
        "                                     num_classes=len(classes))\n",
        "model.sample(x_in_shape=(sequence_size,), seq_lengths_shape=(2,))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_2\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input_6 (InputLayer)            [(None, 8)]          0                                            \n",
            "__________________________________________________________________________________________________\n",
            "embedding_2 (Embedding)         (None, 8, 100)       2987200     input_6[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "input_7 (InputLayer)            [(None, 2)]          0                                            \n",
            "__________________________________________________________________________________________________\n",
            "bidirectional (Bidirectional)   [(None, 8, 128), (No 58624       embedding_2[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "tf_op_layer_Cast_1 (TensorFlowO [(None, 2)]          0           input_7[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "tf_op_layer_GatherNd_1 (TensorF [(None, 128)]        0           bidirectional[0][0]              \n",
            "                                                                 tf_op_layer_Cast_1[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "dense_2 (Dense)                 (None, 100)          12900       tf_op_layer_GatherNd_1[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "dense_3 (Dense)                 (None, 4)            404         dense_2[0][0]                    \n",
            "==================================================================================================\n",
            "Total params: 3,059,128\n",
            "Trainable params: 3,059,128\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "O_XtqY6n6Qdh",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Compile\n",
        "model.compile(optimizer=Adam(lr=LEARNING_RATE),\n",
        "              loss=SparseCategoricalCrossentropy(),\n",
        "              metrics=['accuracy'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "k07ilI7K6N_v",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Callbacks\n",
        "callbacks = [EarlyStopping(monitor='val_loss', patience=EARLY_STOPPING_CRITERIA, verbose=1, mode='min'),\n",
        "             ReduceLROnPlateau(patience=1, factor=0.1, verbose=0),\n",
        "             TensorBoard(log_dir='tensorboard/simple_birnn', histogram_freq=1, update_freq='epoch')]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "uuZwiGRt6X-w",
        "colab_type": "code",
        "outputId": "25bc5ff6-e534-41b3-aaa0-7efd5ff9f1f9",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 306
        }
      },
      "source": [
        "# Training\n",
        "training_history = model.fit_generator(generator=training_generator,\n",
        "                                       epochs=NUM_EPOCHS,\n",
        "                                       validation_data=validation_generator,\n",
        "                                       callbacks=callbacks,\n",
        "                                       shuffle=False,\n",
        "                                       class_weight=class_weights,\n",
        "                                       verbose=1)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "339/339 [==============================] - 58s 172ms/step - loss: 5.5643e-05 - accuracy: 0.3215 - val_loss: 0.7214 - val_accuracy: 0.7438\n",
            "Epoch 2/10\n",
            "339/339 [==============================] - 57s 169ms/step - loss: 2.6350e-05 - accuracy: 0.7730 - val_loss: 0.4666 - val_accuracy: 0.8379\n",
            "Epoch 3/10\n",
            "339/339 [==============================] - 56s 166ms/step - loss: 1.8735e-05 - accuracy: 0.8625 - val_loss: 0.4337 - val_accuracy: 0.8488\n",
            "Epoch 4/10\n",
            "339/339 [==============================] - 57s 167ms/step - loss: 1.5435e-05 - accuracy: 0.8906 - val_loss: 0.4163 - val_accuracy: 0.8547\n",
            "Epoch 5/10\n",
            "339/339 [==============================] - 57s 168ms/step - loss: 1.3246e-05 - accuracy: 0.9098 - val_loss: 0.4095 - val_accuracy: 0.8599\n",
            "Epoch 6/10\n",
            "339/339 [==============================] - 57s 169ms/step - loss: 1.1526e-05 - accuracy: 0.9198 - val_loss: 0.4225 - val_accuracy: 0.8589\n",
            "Epoch 7/10\n",
            "339/339 [==============================] - 57s 167ms/step - loss: 8.7837e-06 - accuracy: 0.9390 - val_loss: 0.4213 - val_accuracy: 0.8627\n",
            "Epoch 8/10\n",
            "339/339 [==============================] - 57s 167ms/step - loss: 8.2887e-06 - accuracy: 0.9423 - val_loss: 0.4217 - val_accuracy: 0.8635\n",
            "Epoch 00008: early stopping\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "jR3trOIQcWY2",
        "colab_type": "text"
      },
      "source": [
        "# Gated RNNs: LSTMs & GRUs"
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "IqFsufNVch4W",
        "colab_type": "text"
      },
      "source": [
        "While our simple RNNs so far are great for sequentially processing our inputs, they have quite a few disadvantages. They commonly suffer from exploding or vanishing gradients as a result using the same set of weights ($W_{xh}$ and $W_{hh}$) with each timestep's input. During backpropagation, this can cause gradients to explode (>1) or vanish (<1). If you multiply any number greater than 1 with itself over and over, it moves towards infinity (exploding gradients) and similarily,  If you multiply any number less than 1 with itself over and over, it moves towards zero (vanishing gradients). To mitigate this issue, gated RNNs were devised to selectively retrain information. If you're interested in learning more of the specifics, this [post](http://colah.github.io/posts/2015-08-Understanding-LSTMs/) is a must-read.\n",
        "\n",
        "There are two popular types of gated RNNs: Long Short-term Memory (LSTMs) units and Gated Recurrent Units (GRUs).\n",
        "\n",
        "<img src=\"https://raw.githubusercontent.com/practicalAI/images/master/images/12_Recurrent_Neural_Networks/gated_rnns.png\" width=\"600\"><br>\n",
        "<a href=\"http://colah.github.io/posts/2015-08-Understanding-LSTMs/\">Understanding LSTM Networks</a> - Chris Olah\n",
        "\n",
        "\n",
        "<div align=\"left\">\n",
        "<img src=\"https://raw.githubusercontent.com/practicalAI/images/master/images/lightbulb.gif\" width=\"45px\" align=\"left\" hspace=\"10px\">\n",
        "</div>\n",
        "\n",
        "When deciding between LSTMs and GRUs, empirical performance is the best factor but in genreal GRUs offer similar perforamnce with less complexity (less weights). \n",
        "\n"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "PqVCqWv2fz4N",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "from tensorflow.keras.layers import GRU"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "7tBHovnxcTwV",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# RNN cell\n",
        "gru = GRU(units=RNN_HIDDEN_DIM, \n",
        "          dropout=DROPOUT_P, \n",
        "          recurrent_dropout=RNN_DROPOUT_P,\n",
        "          return_sequences=True,\n",
        "          return_state=True) "
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "n7BfnoAbcTtk",
        "colab_type": "code",
        "outputId": "0aa920b0-0e17-4257-929c-599d74882ac7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 442
        }
      },
      "source": [
        "model = TextClassificationBiRNNModel(vocab_size=vocab_size,\n",
        "                                     embedding_dim=EMBEDDING_DIM,\n",
        "                                     rnn_cell=gru,\n",
        "                                     hidden_dim=HIDDEN_DIM,\n",
        "                                     dropout_p=DROPOUT_P,\n",
        "                                     num_classes=len(classes))\n",
        "model.sample(x_in_shape=(sequence_size,), seq_lengths_shape=(2,))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Model: \"model_3\"\n",
            "__________________________________________________________________________________________________\n",
            "Layer (type)                    Output Shape         Param #     Connected to                     \n",
            "==================================================================================================\n",
            "input_8 (InputLayer)            [(None, 8)]          0                                            \n",
            "__________________________________________________________________________________________________\n",
            "embedding_3 (Embedding)         (None, 8, 100)       2987200     input_8[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "input_9 (InputLayer)            [(None, 2)]          0                                            \n",
            "__________________________________________________________________________________________________\n",
            "bidirectional_1 (Bidirectional) [(None, 8, 128), (No 176640      embedding_3[0][0]                \n",
            "__________________________________________________________________________________________________\n",
            "tf_op_layer_Cast_2 (TensorFlowO [(None, 2)]          0           input_9[0][0]                    \n",
            "__________________________________________________________________________________________________\n",
            "tf_op_layer_GatherNd_2 (TensorF [(None, 128)]        0           bidirectional_1[0][0]            \n",
            "                                                                 tf_op_layer_Cast_2[0][0]         \n",
            "__________________________________________________________________________________________________\n",
            "dense_4 (Dense)                 (None, 100)          12900       tf_op_layer_GatherNd_2[0][0]     \n",
            "__________________________________________________________________________________________________\n",
            "dense_5 (Dense)                 (None, 4)            404         dense_4[0][0]                    \n",
            "==================================================================================================\n",
            "Total params: 3,177,144\n",
            "Trainable params: 3,177,144\n",
            "Non-trainable params: 0\n",
            "__________________________________________________________________________________________________\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Vd8SL8UIcTqc",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Compile\n",
        "model.compile(optimizer=Adam(lr=LEARNING_RATE),\n",
        "              loss=SparseCategoricalCrossentropy(),\n",
        "              metrics=['accuracy'])"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "c8dNn6VPcTkx",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Callbacks\n",
        "callbacks = [EarlyStopping(monitor='val_loss', patience=EARLY_STOPPING_CRITERIA, verbose=1, mode='min'),\n",
        "             ReduceLROnPlateau(patience=1, factor=0.1, verbose=0),\n",
        "             TensorBoard(log_dir='tensorboard/gru', histogram_freq=1, update_freq='epoch')]"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "NbpjRVficTcY",
        "colab_type": "code",
        "outputId": "db96b684-6b21-415e-cebf-b755210cbcf2",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 340
        }
      },
      "source": [
        "# Training\n",
        "training_history = model.fit_generator(generator=training_generator,\n",
        "                                       epochs=NUM_EPOCHS,\n",
        "                                       validation_data=validation_generator,\n",
        "                                       callbacks=callbacks,\n",
        "                                       shuffle=False,\n",
        "                                       class_weight=class_weights,\n",
        "                                       verbose=1)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "Epoch 1/10\n",
            "339/339 [==============================] - 107s 315ms/step - loss: 6.3408e-05 - accuracy: 0.2798 - val_loss: 1.3476 - val_accuracy: 0.4068\n",
            "Epoch 2/10\n",
            "339/339 [==============================] - 107s 314ms/step - loss: 5.0585e-05 - accuracy: 0.4524 - val_loss: 0.6740 - val_accuracy: 0.7573\n",
            "Epoch 3/10\n",
            "339/339 [==============================] - 106s 311ms/step - loss: 2.4130e-05 - accuracy: 0.7985 - val_loss: 0.4754 - val_accuracy: 0.8327\n",
            "Epoch 4/10\n",
            "339/339 [==============================] - 106s 312ms/step - loss: 1.7806e-05 - accuracy: 0.8694 - val_loss: 0.4262 - val_accuracy: 0.8508\n",
            "Epoch 5/10\n",
            "339/339 [==============================] - 106s 314ms/step - loss: 1.4815e-05 - accuracy: 0.8978 - val_loss: 0.4231 - val_accuracy: 0.8530\n",
            "Epoch 6/10\n",
            "339/339 [==============================] - 106s 312ms/step - loss: 1.2935e-05 - accuracy: 0.9117 - val_loss: 0.4168 - val_accuracy: 0.8563\n",
            "Epoch 7/10\n",
            "339/339 [==============================] - 107s 314ms/step - loss: 1.1493e-05 - accuracy: 0.9233 - val_loss: 0.4288 - val_accuracy: 0.8532\n",
            "Epoch 8/10\n",
            "339/339 [==============================] - 107s 315ms/step - loss: 9.1927e-06 - accuracy: 0.9365 - val_loss: 0.4204 - val_accuracy: 0.8600\n",
            "Epoch 9/10\n",
            "339/339 [==============================] - 107s 315ms/step - loss: 8.7459e-06 - accuracy: 0.9411 - val_loss: 0.4198 - val_accuracy: 0.8599\n",
            "Epoch 00009: early stopping\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "4ytG83xxcTZp",
        "colab_type": "code",
        "outputId": "5cb94c46-9189-4474-efa2-f9ec42e592e5",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Evaluation\n",
        "testing_history = model.evaluate_generator(generator=testing_generator, \n",
        "                                           verbose=1)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "71/71 [==============================] - 8s 117ms/step - loss: 0.3997 - accuracy: 0.8634\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "0eXhh4rqcTXE",
        "colab_type": "code",
        "outputId": "c30a94d4-ac30-46d2-e711-7a8912408557",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "%tensorboard --logdir tensorboard"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "Reusing TensorBoard on port 6006 (pid 927), started 0:23:43 ago. (Use '!kill 927' to kill it.)"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "display_data",
          "data": {
            "text/html": [
              "\n",
              "    <div id=\"root\"></div>\n",
              "    <script>\n",
              "      (function() {\n",
              "        window.TENSORBOARD_ENV = window.TENSORBOARD_ENV || {};\n",
              "        window.TENSORBOARD_ENV[\"IN_COLAB\"] = true;\n",
              "        document.querySelector(\"base\").href = \"https://localhost:6006\";\n",
              "        function fixUpTensorboard(root) {\n",
              "          const tftb = root.querySelector(\"tf-tensorboard\");\n",
              "          // Disable the fragment manipulation behavior in Colab. Not\n",
              "          // only is the behavior not useful (as the iframe's location\n",
              "          // is not visible to the user), it causes TensorBoard's usage\n",
              "          // of `window.replace` to navigate away from the page and to\n",
              "          // the `localhost:<port>` URL specified by the base URI, which\n",
              "          // in turn causes the frame to (likely) crash.\n",
              "          tftb.removeAttribute(\"use-hash\");\n",
              "        }\n",
              "        function executeAllScripts(root) {\n",
              "          // When `script` elements are inserted into the DOM by\n",
              "          // assigning to an element's `innerHTML`, the scripts are not\n",
              "          // executed. Thus, we manually re-insert these scripts so that\n",
              "          // TensorBoard can initialize itself.\n",
              "          for (const script of root.querySelectorAll(\"script\")) {\n",
              "            const newScript = document.createElement(\"script\");\n",
              "            newScript.type = script.type;\n",
              "            newScript.textContent = script.textContent;\n",
              "            root.appendChild(newScript);\n",
              "            script.remove();\n",
              "          }\n",
              "        }\n",
              "        function setHeight(root, height) {\n",
              "          // We set the height dynamically after the TensorBoard UI has\n",
              "          // been initialized. This avoids an intermediate state in\n",
              "          // which the container plus the UI become taller than the\n",
              "          // final width and cause the Colab output frame to be\n",
              "          // permanently resized, eventually leading to an empty\n",
              "          // vertical gap below the TensorBoard UI. It's not clear\n",
              "          // exactly what causes this problematic intermediate state,\n",
              "          // but setting the height late seems to fix it.\n",
              "          root.style.height = `${height}px`;\n",
              "        }\n",
              "        const root = document.getElementById(\"root\");\n",
              "        fetch(\".\")\n",
              "          .then((x) => x.text())\n",
              "          .then((html) => void (root.innerHTML = html))\n",
              "          .then(() => fixUpTensorboard(root))\n",
              "          .then(() => executeAllScripts(root))\n",
              "          .then(() => setHeight(root, 800));\n",
              "      })();\n",
              "    </script>\n",
              "  "
            ],
            "text/plain": [
              "<IPython.core.display.HTML object>"
            ]
          },
          "metadata": {
            "tags": []
          }
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "vskwiiI3V3S6",
        "colab_type": "text"
      },
      "source": [
        "# Evaluation"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "Itq7lT9qV9Y8",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import io\n",
        "import itertools\n",
        "import json\n",
        "import matplotlib.pyplot as plt\n",
        "from sklearn.metrics import classification_report\n",
        "from sklearn.metrics import confusion_matrix\n",
        "from sklearn.metrics import precision_recall_fscore_support"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "175J-ELiV-I6",
        "colab_type": "text"
      },
      "source": [
        "### Components"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "VT9C8rUyWAWF",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def plot_confusion_matrix(y_true, y_pred, classes, cmap=plt.cm.Blues):\n",
        "    \"\"\"Plot a confusion matrix using ground truth and predictions.\"\"\"\n",
        "    # Confusion matrix\n",
        "    cm = confusion_matrix(y_true, y_pred)\n",
        "    cm_norm = cm.astype('float') / cm.sum(axis=1)[:, np.newaxis]\n",
        "\n",
        "    #  Figure\n",
        "    fig = plt.figure()\n",
        "    ax = fig.add_subplot(111)\n",
        "    cax = ax.matshow(cm, cmap=plt.cm.Blues)\n",
        "    fig.colorbar(cax)\n",
        "\n",
        "    # Axis\n",
        "    plt.title(\"Confusion matrix\")\n",
        "    plt.ylabel(\"True label\")\n",
        "    plt.xlabel(\"Predicted label\")\n",
        "    ax.set_xticklabels([''] + classes)\n",
        "    ax.set_yticklabels([''] + classes)\n",
        "    ax.xaxis.set_label_position('bottom') \n",
        "    ax.xaxis.tick_bottom()\n",
        "\n",
        "    # Values\n",
        "    thresh = cm.max() / 2.\n",
        "    for i, j in itertools.product(range(cm.shape[0]), range(cm.shape[1])):\n",
        "        plt.text(j, i, f\"{cm[i, j]:d} ({cm_norm[i, j]*100:.1f}%)\",\n",
        "                 horizontalalignment=\"center\",\n",
        "                 color=\"white\" if cm[i, j] > thresh else \"black\")\n",
        "\n",
        "    # Display\n",
        "    plt.show()"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "OyguCCS3W6DO",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_performance(y_true, y_pred, classes):\n",
        "    \"\"\"Per-class performance metrics.\"\"\"\n",
        "    performance = {'overall': {}, 'class': {}}\n",
        "    y_pred = np.argmax(y_pred, axis=1)\n",
        "    metrics = precision_recall_fscore_support(y_true, y_pred)\n",
        "\n",
        "    # Overall performance\n",
        "    performance['overall']['precision'] = np.mean(metrics[0])\n",
        "    performance['overall']['recall'] = np.mean(metrics[1])\n",
        "    performance['overall']['f1'] = np.mean(metrics[2])\n",
        "    performance['overall']['num_samples'] = np.float64(np.sum(metrics[3]))\n",
        "\n",
        "    # Per-class performance\n",
        "    for i in range(len(classes)):\n",
        "        performance['class'][classes[i]] = {\n",
        "            \"precision\": metrics[0][i],\n",
        "            \"recall\": metrics[1][i],\n",
        "            \"f1\": metrics[2][i],\n",
        "            \"num_samples\": np.float64(metrics[3][i])\n",
        "        }\n",
        "\n",
        "    return performance"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yCAHPsyWWAar",
        "colab_type": "text"
      },
      "source": [
        "### Operations"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "MHfGZh7JZGe6",
        "colab_type": "code",
        "outputId": "211b0e00-2ee5-4c6a-a49f-29fca1d908cd",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 68
        }
      },
      "source": [
        "# Evaluation\n",
        "test_history = model.evaluate_generator(generator=testing_generator, verbose=1)\n",
        "y_pred = model.predict_generator(generator=testing_generator, verbose=1)\n",
        "print (f\"test history: {test_history}\")"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "71/71 [==============================] - 8s 117ms/step - loss: 0.3997 - accuracy: 0.8634\n",
            "71/71 [==============================] - 8s 113ms/step\n",
            "test history: [0.3996739597387717, 0.8633889]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "qdAj6KyCU88E",
        "colab_type": "code",
        "outputId": "fe62cde7-1111-4518-9002-45d4c700c4a6",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 595
        }
      },
      "source": [
        "# Class performance\n",
        "performance = get_performance(y_true=y_test,\n",
        "                              y_pred=y_pred,\n",
        "                              classes=classes)\n",
        "print (json.dumps(performance, indent=4))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "{\n",
            "    \"overall\": {\n",
            "        \"precision\": 0.8634842590676457,\n",
            "        \"recall\": 0.863388888888889,\n",
            "        \"f1\": 0.8631243801644576,\n",
            "        \"num_samples\": 18000.0\n",
            "    },\n",
            "    \"class\": {\n",
            "        \"Business\": {\n",
            "            \"precision\": 0.8325922632281014,\n",
            "            \"recall\": 0.8322222222222222,\n",
            "            \"f1\": 0.8324072016003555,\n",
            "            \"num_samples\": 4500.0\n",
            "        },\n",
            "        \"Sci/Tech\": {\n",
            "            \"precision\": 0.8433949654711517,\n",
            "            \"recall\": 0.8413333333333334,\n",
            "            \"f1\": 0.8423628879741907,\n",
            "            \"num_samples\": 4500.0\n",
            "        },\n",
            "        \"Sports\": {\n",
            "            \"precision\": 0.8840885142255005,\n",
            "            \"recall\": 0.9322222222222222,\n",
            "            \"f1\": 0.9075175770686857,\n",
            "            \"num_samples\": 4500.0\n",
            "        },\n",
            "        \"World\": {\n",
            "            \"precision\": 0.8938612933458294,\n",
            "            \"recall\": 0.8477777777777777,\n",
            "            \"f1\": 0.8702098540145985,\n",
            "            \"num_samples\": 4500.0\n",
            "        }\n",
            "    }\n",
            "}\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "nRbPfqgZWaof",
        "colab_type": "code",
        "outputId": "4e1da3b1-fc78-48f6-a0c6-bf58cab626c1",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 598
        }
      },
      "source": [
        "# Confusion matrix\n",
        "plt.rcParams[\"figure.figsize\"] = (7,7)\n",
        "y_pred = np.argmax(y_pred, axis=1)\n",
        "plot_confusion_matrix(y_test, y_pred, classes=classes)\n",
        "print (classification_report(y_test, y_pred))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "display_data",
          "data": {
            "image/png": "iVBORw0KGgoAAAANSUhEUgAAAc8AAAGKCAYAAABq7cr0AAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAADh0RVh0U29mdHdhcmUAbWF0cGxvdGxpYiB2ZXJzaW9uMy4xLjEsIGh0\ndHA6Ly9tYXRwbG90bGliLm9yZy8QZhcZAAAgAElEQVR4nOzdd3gUVRfA4d/ZJEDoXTrSO4QWQu9V\nEFAQBGmCiB+oWCmiiIKKBRFUEEFFULr0XkQ6CR1BOii9V5NAyv3+mMmSkE3DTQjkvD77sHPb3BmT\nnL137syKMQallFJKxZ/jQXdAKaWUetho8FRKKaUSSIOnUkoplUAaPJVSSqkE0uCplFJKJZDng+6A\nUkqpR4dHxoLGhAa5pS0TdHG5MaaZWxpzMw2eSiml3MaEBpG6xDNuaSt41zfZ3dJQItDgqZRSyo0E\n5NG/IvjoH6FSSinlZho8lYqDiHiLyEIRuS4is/5DO51FZIU7+/agiEhtETn4oPuhkiEBRNzzSsY0\neKpHhoh0EpFtInJLRM6KyFIRqeWGptsBjwHZjDHt77cRY8wvxpgmbuhPohIRIyJFYytjjFlvjCmR\nVH1SDxlxuOeVjCXv3ikVTyLyOjAa+Agr0BUAvgVau6H5gsAhY0yoG9p66ImIrpVQKZ4GT/XQE5FM\nwAdAX2PMb8aYf40xIcaYhcaYt+wyqUVktIicsV+jRSS1nVdPRE6JyBsicsEetfaw84YB7wEd7BFt\nTxF5X0SmRtr/4/ZozdPe7i4ix0TkpogcF5HOkdI3RKpXQ0QC7OngABGpESlvrYh8KCIb7XZWiIjL\nlYeR+v92pP63EZEWInJIRK6IyOBI5X1FZLOIXLPLfi0iqey8dXax3fbxdojU/gAROQf8GJFm1yli\n76OSvZ1HRC6KSL3/9D9WPbx02laph0J1IA0wN5Yy7wB+gA9QAfAFhkTKzwVkAvICPYFvRCSLMWYo\n1mh2hjEmvTFmUmwdEZF0wBiguTEmA1AD2OWiXFZgsV02GzAKWCwi2SIV6wT0AHICqYA3Y9l1Lqxz\nkBcr2H8PPAdUBmoD74pIIbtsGPAakB3r3DUE/gdgjKljl6lgH++MSO1nxRqF9468Y2PMUWAAMFVE\n0gI/ApONMWtj6a96ZIlO2yr1kMgGXIpjWrUz8IEx5oIx5iIwDOgSKT/Ezg8xxiwBbgH3e00vHCgr\nIt7GmLPGmH0uyjwBHDbGTDHGhBpjpgEHgFaRyvxojDlkjAkCZmIF/piEACOMMSHAdKzA+JUx5qa9\n//1YHxowxmw3xmyx93sC+A6oG49jGmqMuW33JwpjzPfAEWArkBvrw4pSjywNnupRcBnIHse1uDzA\n35G2/7bTnG3cE3wDgfQJ7Ygx5l+gA9AHOCsii0WkZDz6E9GnvJG2zyWgP5eNMWH2+4jgdj5SflBE\nfREpLiKLROSciNzAGlnHdTP6RWNMcBxlvgfKAmONMbfjKKseZTptq9RDYTNwG2gTS5kzWFOOEQrY\naffjXyBtpO1ckTONMcuNMY2xRmAHsIJKXP2J6NPp++xTQozD6lcxY0xGYDDWDQaxMbFlikh6rAVb\nk4D37WlplRIJOm2r1MPAGHMd6zrfN/ZCmbQi4iUizUXkU7vYNGCIiOSwF968B0yNqc047ALqiEgB\ne7HSoIgMEXlMRFrb1z5vY03/hrtoYwlQ3L69xlNEOgClgUX32aeEyADcAG7Zo+KX7sk/DxROYJtf\nAduMMb2wruWO/8+9VCoZ0+CpHgnGmC+A17EWAV0ETgL9gHl2keHANmAPsBfYYafdz75WAjPstrYT\nNeA57H6cAa5gXUu8NzhhjLkMtATewJp2fhtoaYy5dD99SqA3sRYj3cQaFc+4J/99YLK9GjfOh5SK\nSGugGXeP83WgUsQqY5XSuGnKNplP24oxsc7GKKWUUvHmSJ/bpC7fwy1tBW/+eLsxpopbGnMzHXkq\npZR6aImIh4jsFJFF9nYhEdkqIkdEZEake5hT29tH7PzHI7UxyE4/KCJN47NfDZ5KKaXcK2mnbV8F\n/oq0PRL40hhTFLiKdd829r9X7fQv7XKISGmgI1AG6/LDtyLiEddONXgqpZRyo6R7SIKI5MO6Z3qi\nvS1AA2C2XWQyd1fht7a3sfMb2uVbA9Pte5iPY92v7BvXvjV4KqWUSq6yi/VlDxGv3vfkj8ZabBex\noj0bcC3SPdunuHvvdF6shYTY+dft8s50F3VipA94Vkop5T4RX0nmHpdiWjAkIi2BC8aY7Q/iOcoa\nPJVSSrlX0jzgoCbwpIi0wHquc0as+40zi4inPbrMx90Hj5wG8gOn7KeRZcK6TSwiPULkOjHSaVul\nlFIPHWPMIGNMPmPM41gLftYYYzoDv2N9By9AN2C+/X6BvY2dv8ZY92ouADraq3ELAcUA/7j2ryNP\npZRSbiQP+tF6A4DpIjIc2In1yEjsf6eIyBGsB5h0BDDG7BORmVhfnhCK9dWGYdGbjUofkqCUUspt\nHBnymtRV+rilreC17+lDEpRSSqlHhU7bKqWUcp+Ib1V5xGnwVEop5V7J/KHu7vDofzxQSiml3ExH\nnkoppdzoga+2TRIaPJVSSrmXTtsqpZRS6l468lRKKeVeOm2rlFJKJUDCvovzoaXBUymllHvpyFMB\nSKr0RtJme9DdeOiUK5T9QXfhoeThePQ/tavk45+/T3Dp0iX9oUsgDZ7xIGmzkbr2wAfdjYfOsqk9\nH3QXHkoZ0uivpUo6dWr4ur9RnbZVSimlEiJl3Of56B+hUkop5WY68lRKKeVeOm2rlFJKJUAK+VaV\nR/8IlVJKKTfTkadSSik3ShkLhjR4KqWUcq8UcM3z0f94oJRSSrmZjjyVUkq5l07bKqWUUgmk07ZK\nKaWUupeOPJVSSrmP6GpbpZRSKuF02lYppZRS99KRp1JKKbeSFDDy1OCplFLKbYSUETx12lYppZRK\nIB15KqWUch+xX484DZ5KKaXcSHTaVimllFLR6chTKaWUW6WEkacGT6WUUm6VEoKnTtsqpZRSCaQj\nT6WUUm6VEkaeGjyVUkq5Twq5VUWnbZVSSqkE0uCplFLKbcS+z9Mdr1j3I5JGRPxFZLeI7BORYXb6\nTyJyXER22S8fO11EZIyIHBGRPSJSKVJb3UTksP3qFp/j1GlbpZRSbpVE1zxvAw2MMbdExAvYICJL\n7by3jDGz7ynfHChmv6oB44BqIpIVGApUAQywXUQWGGOuxrZzHXkqpZR66BjLLXvTy36ZWKq0Bn62\n620BMotIbqApsNIYc8UOmCuBZnHtX4OnUkopt3LjtG12EdkW6dX7nv14iMgu4AJWANxqZ42wp2a/\nFJHUdlpe4GSk6qfstJjSY6XTtkoppdzKjdO2l4wxVWLKNMaEAT4ikhmYKyJlgUHAOSAVMAEYAHzg\nrg5F0JHnfUrt5cH6L55i65j2bP+mA0M6VXXmrfqkDVu+as+Wr9pz7KeuzHwn6gxA5WI5uDnvRdrW\nKOxMuzXvRWedWUOax7jfz3rVpGaZ3ADUK5+XTaPbseWr9qwe2YbCuTMC0KtZaQLGPuNML5k/S7R2\n8mVPx7IRT7Ljmw5s/6YDfVuVc+YN7+aH/5hnmPhaA2dax3rF6Pdkeed2mYJZmdC/fnxPV7yFhYXR\nuLYvXTu0cab1faEbtaqUpX71irzWtzchISEALFu8gIY1KtOoVlWa1avO1s0bXbYZFBTEUy0aERYW\nZh3f0MHUr16R+tUrMv+3WbH2Z/H8ueTJnJrdO7cD4L9lEw1rVKZZveocO3oYgOvXrtGxbQvCw8Od\n9Z5p3Yxr12K9ZOI2L/XuSaH8ufCtVD5K+kcfDqN44fzU8K1EDd9KLF+2BIBtAf7OtOpVK7Jg/lyX\n7RpjeKJpI27cuEFwcDD1avlRvWpFqlYsx4gP3o+1T/PnziFDGg92bN8GwOZNG/Gr4kOdGr4cOWKd\nt2vXrtH6iaZRzlur5k24evXBnrduz3V0np8yxQtTw9daV3Lnzh36vPA81SpXoHrViqz/Y22MbT/3\nbHuOHzsWJe2Zp1tH21cEYwxvvf4qFUoXx6+KD7t27gDg0KGD1K5eFb8qPmzdshmA0NBQWjVvQmBg\noLN+9y7POs9rSmSMuQb8DjQzxpy1p2ZvAz8Cvnax00D+SNXy2WkxpcdKg+d9uh0SRrN3FlDtlVlU\ne2UWTSrlx7fEYwA0GjgPv1dn4ffqLLYePM+8TXd/iRwOYXi36qzaeTJKe0F3wpx12g9fiitZM6TG\nt8RjbNx3FoAx/6tDj89X4ffqLGb8cZiBHSoDMOOPw1R9eSZ+r85i1JxdjOxZI1pboWGGgT9solLf\nGdR98zdefKIsJfNnIWPaVPgUyY7vKzO5ExpOmYJZSZPKg66NSjJ+8Z/O+vv+vkLebOnJnyP9fzuR\n95g4bizFSpSMkvZU+46sD9jLmk07CA4O4teffwCgdt0GrNq4jVUbAhj19QTefKWPyzanT/2JFq1a\n4+HhwarlS9i7eycr1weweNUGxo/9kps3brisd+vmTSaO/5pKVXydad99PZops+Yz7OPPmfLD9wCM\n/vxjXnl9AA7H3V+ndh06M3nid//pXMRX5y7dmLtgicu8vi/3Z5P/Djb576BpsxYAlC5TlnWb/Nnk\nv4O5C5bwar+XCA0NjVZ3+bIllCtfnowZM5I6dWoWLVvF5oCdbPLfwaqVy/HfusXlPm/evMm3X4+l\nim81Z9rYr0YxZ94iPvlsFJO+t87Lp5+M4M23B0U5bx07dWbid+Pu+1wkREznbfLU6c5z9mTbp3iy\ndVsAfvphIgBbt+9mweLlDB74VpTAH+Gv/fsICwujUOG7H47nz/uN9Oli/l1ZsXwpR48cZte+g4z5\nZjyvvdIXgB8mTmDk518yZ94ixoz+AoCJE8bT4dlOpE2b1lm/1wt9GP3FZ/dxFhKBuPEV225Ectgj\nTkTEG2gMHLCvYyLW8LcNEPGHawHQ1V516wdcN8acBZYDTUQki4hkAZrYabHS4Pkf/Bts/cHx8nTg\n6enAmKjXqjN4e1G3fF4WbjnuTPtfy3LM23SUi9eDEry/NjUKs2LHP85tYyBj2lSA9e/Zy9Yn0ZtB\nIc4y6dJ4uryCfu5qILuOXgLgVlAIB05eJU+2dIQbg5en9WORNrUnIWHh9G/rw7hFewkNi/qHYon/\nCdrXLprg44jJmdOnWL1iKZ269IiS3rBJc+c1kIqVqnD2jPWhMF369M7pocDAf2OcKvpt1nSatmgF\nwKGDf+FXozaenp6kTZeOUmXK8fvqFS7rfTriffr2f4PUqdM40zy9vAgKCiQoKBBPLy9OHD/KmdOn\nqFG7bpS6TVq0ZN6cGfd3IhKoVu06ZMmSNd7l06ZNi6endcUmODg4xvM2c9qvPNHyScCahkuf3vrj\nHxISQkhISIz1hg97j9fefIs0kc6bl5cXgUGBBAUG4uXlxbGjRzl96iS169aLUrdFyyeZNXN6vI/l\nv4jrvBljmDt7Fu06dATgwF/7qVvPmm3JkTMnmTJldo6sI5sR6bwB3Lp1i6+/Gs3bg96JcV+LFy7g\n2c5dEBF8q/lx7do1zp09i5eXF0GBgQQGWeft2rVrLF28kE7PdY1Sv0at2qxds9rlh6AHISluVQFy\nA7+LyB4gAOua5yLgFxHZC+wFsgPD7fJLgGPAEeB74H8AxpgrwId2GwHAB3ZarBIteIpImH2PzW4R\n2SEi0Yc/8Wunj4h0jbtk0nM4hC1fteefKd1Zs/MUAYcuRMlv5VeItbtPOYNZnqzpeLJ6ISYs3Ret\nrTSpPNgw6mn++OwpWvk97nJ/1UvlZueRi87t/41dy9yhT3Dkxy50ql+cz2fvcOa92KIM+yZ0YkT3\n6rzx3YZYj6NAzgz4FMlOwMHz3AoKYfm2f9jyVXvOXfmXG//eoWqJnCzcciJavR1HLlLDnkJ2h6GD\n3mTIBx9HGYlEFhISwuwZv1K/YRNn2tKF86ldtRxdn2nDqK8nRKtz584d/jlxnPwFHwegdNny/L56\nBYGBgVy+fIlN69dy5tTJaPX27NrJmdOnaNS0RZT0l197m1f79OTrUZ/R44WX+OTDoQwY8n60+pkz\nZ+HO7dtcuXI5AWfA/SaM+wa/Kj681LtnlOnQAP+tVK1YDr8qFRg99ltnMI1sy+ZN+FSq7NwOCwuj\nhm8lCufPRf2GjagaaWQZYdfOHZw6dZJmzZ+Ikv7GWwPp3bM7X3w2khf79OWD94fw7vsfRqufJYt1\n3i5ffrDnDWDjhvXkfOwxihYtBkDZcuVZsnghoaGhnDh+nF07t3Paxc/Ols2bqBjpvA0f9h4v938N\nb++00cpGOHPmNHnz3Z05zJs3H2fOnKb3i//j808/4cVePXjz7UGM/Hh4tNE6gMPhoHCRIuzds/u/\nHvZDwxizxxhT0RhT3hhT1hjzgZ3ewBhTzk57LmJFrj2V29cYU8TO3xaprR+MMUXt14/x2X9ijjyD\njDE+xpgKWBdwP76fRowx440xP7u3a+4RHm7we3UWRXv8TJXiOSldIOqn2GfqFmPmuiPO7c9eqMmQ\nn7ZgXAwFSzw/lVqvz6Hb5yv5rFdNCuXKGK1MrqxpuXQj2Ln9cuvytB22mKI9pjBl1UFG9qrpzPtu\nyT7K9P6VIZO3OKdzXUmXxpNpg5ry1vcbnUF+1G+78Ht1FgN/2Mx7z/ny4S8BdG9SiqkDGjPgGed9\nxVy4FkTurOniPlHxsHLZYrLnyEF5n0oxlhn0xiv41ahFtRq1nGnNW7VmfcBefvhlFp+OeD9anSuX\nL5ExUybndr0GjWnYuBlPNqnL/3p2obKvHx4eHlHqhIeHM+ydtxk6fGS09sqWr8CiVeuZvWgF/5w4\nzmOP5cIYw4s9OtOvd3cuXjjvLJstR07Onz2bkNPgVr1692HPX4fZ5L+DXLlyM3jAm868qr7VCNi5\nl7UbtzLqs5EEBwdHq3/16hUyZMjg3Pbw8GCT/w4OHP2H7QEB7N/3Z5Ty4eHhDHr7TT765PNobZWv\n4MPv6zaxZMVqjh8/Rq5cuTHG0O25jvTq3oUL5++et+w5cnLu7Bl3nIL/ZPbM6bR7pqNzu2v358mb\nNx91avgy4K3XqOZXHcc9PzsA586dJXv2HADs2b2LY8eOOqd+Eyp/gQIsXbmGNX9sxDttWs6cPkWJ\nkqV4oUdXuj3XkcOHDznL5siRk7PJ4Lwl1UMSHrSkmrbNCFwFEJF6IrIoIkNEvhaR7vb7T0Rkv73E\n+HM77X0RedN+v1ZERor1VIlDIlLbTvcQkc9EJMCu+6KdnltE1tkj4D9FpLZd9id7e6+IvPZfD+76\nv3f4Y+9pmlS++8kxW8Y0VCmWk6UBfzvTKhXLwc9vNeLAxM60rVGE0S/VcY4yz1z5F4AT52+y7s8z\n+BTOHm0/QbdDSe1l/bJmz5iGcoWyOUe7szccwa/kY9HqzFx3OMaRrKeHg2mDmjJj7SHmbz4eLb9C\n4ewIcOjUNZ6qWZjnRq6kcO5MFMltBaM0qTwIvuOeaaKArZtZsXQxvuWK81LPLmxYt5Z+vbs787/4\nZDiXL13k/Y9cX9fxq1mbf04c5/LlS1HS03h7czv4dpS0V98cyKoNAcyYtxRjDIXtkUWEWzdvcuCv\nfTzdsgm+5YqzY9tWuj/7tHPREFhTeqM//5j+bw9m1MgRvDvsIzp3fZ5J333jLHM7OJg03ml4UHI+\n9hgeHh44HA66P9+L7dsCopUpWbIU6dKljxYIATw9PV1e08ucOTN16tZj5Yqol4Vu3rzJ/v1/0qJJ\nA8oUL0yA/xY6tGsTZWrTGMNnn4zg7UFD+GT4B3w4YiTdn+/FuG/GOsvcvh1MGm/v/3Lo/1loaCgL\n5s/l6XbPONM8PT355LNRbPLfwYzZ87h2/TrFihWPVtfb29v5YcR/62Z27thOmeKFadKwDkcOH6J5\n4wbR6uTJkzfKKPb06VPkyRP1bokPhlqj9XHfjKVbj558OGIknwy/u4g0+HYw3mke7HmLoMHzv/G2\ng9YBYCLWnHKMRCQb0BYoY4wpz9156nt5GmN8gf5YT4UA6Il18bcqUBV4QUQKAZ2A5cYYH6ACsAvw\nAfLaQ/pyWKuxXPWnt9j3Fpk7t6LlZ8+YhkzprOuNaVJ50NAnPwdPXXPmt61RmKUBf3M7JMyZVqrX\nL5S0X3M3HaX/uHUs3HKCzOlSkcq+zpgtYxqql8rFXyejrzg8eOoqRfJYgevqrdtkTJeKovZ2A598\nzv1HBDeA5lUKcuTMdZcncvwr9Th48hpj5u9xmf/ec1X54Bd/vDwdeNjTROHhhrSprSm+Ynkzs+/v\nOC8NxMvgocPZvv8Y/nsPMW7SFGrVqcfXE34C4Jeff2DtmpV8O2lKlOmq48eOOK8z79m1kzt37pA1\na7Yo7WbOnIWw8DDnH7OwsDDnVOr+P/fy17691G3QOEqdjJkyse/YGfz3HsJ/7yEqVanGT9PmUKHi\n3RH8rGlTadi4GVmyZCUoMBBxOHA4HATZKyCNMVy8cJ78BR53y/m5H+cijXoXLphH6TJlADhx/Ljz\n2tg/f//NoUMHKGBPa0dWtHgJjh+3FrtdvHiRa9esn6+goCDWrF5F8RIlopTPlCkTf5++wL5Dx9h3\n6BhVff2YMXselSrfvdPg16k/06RZC7JmzUpgUCAOhwNxOAgKunvezp8/R0EX/UlKv69ZRfHiJcmb\nL58zLTAwkH//tT7krlm1Ek8PT0qWKh2tbomSJTl21Jpx6tX7JQ4fP8W+Q8dYsXodRYsVZ+nKNdHq\ntGjZimm/TMEYg//WLWTKlIlcue9eEtmw7g9y585D0aLFCAq6+/MWGHR3xe2Rw4cpXaas286Bil1i\n3ucZZActRKQ68LN9D05MrgPBwCR7ZLoohnK/2f9uBx633zcByotIO3s7E9YjmAKAH8R6dNM8Y8wu\nETkGFBaRscBiwOVqEWPMBKx7hHBkLhhtojVX1rR8378BHg4HDocwZ8ORKKPM9nWK8vnsnbEc7l0l\n82dhbN+6hBuDQ4TPZ+/kgIvguSzgH3o2K81PK/4iLNzQd+wfTBvUlHBjuHbrNi9+9TsAL7UsS32f\nfISEhnPt1m1eGG39subOmpZvX65H22FLqFE6F50blGDv8cts+ao9AEN/3sry7daCpFZ+j7Pj8EXO\nXrF+Ofccv0TA2Gf488Rl9p6wgk/dcnlYFumYE8vA1/qRL38BWjWuA0CLVm14fcA7LF4wj9nTp+Lp\n6YW3tzfjfpjq8tNq3fqN8N+ykTr1GhISEkLb5tYn/wwZMjL2u5+c1/s+HTGMChUrORcXxSQwMJCZ\nv05h2tzFALzY91W6PNMaL69UfDNxMgB7du2gUhVfl9cS3a1Hl06sX/8Hly9dokSRAgweMpRuPXry\n7uAB7NmzGxGhQMGCjPl6PACbN21g1Oef4uXlhcPhYNRXX5M9e/SZjmbNWrD+j7UUKVKU8+fO8mKv\nHoSFhREeHs5TT7eneYuWAAwfNpSKlStHWSTjSmBgIL9M+Zn5i5cB0O+V13i6TUtSpUrFpMlTAdi5\nYztVfas90PMGMHvmDNp36BCl/MULF2jTqjkOh4M8efLy/Q+TXbbbtFkL1q/7g/oNG8W6/0nfW/8/\ner7Qh6bNWrBi2VIqlC6Od9q0jJswyVnOGMOnn4zgp6nWQqruPV+gV/cuhIaG8uUYa6bjwvnzeHt7\n81iuXPd3MtwteQ8a3ULuXSHqtoZFbhlj0kfaPg+UA4oDg40xLez0icAGY8xPYj0JoiHQDnjcGNNA\nRN4HbhljPheRtcCbxphtIpId2GaMeVxE5gATjDHRlheLSB7gCaAvMMoY87OIpMd6JFMX4Iox5vnY\njsWRuaBJXXvgfzwj7rF6ZBue+mAJ1/+980D7kcrTwcqP29BgwFzCwl3/DB2b2jOJe+Xanl07+f7b\nMYydEK91AG7x7oDXadKiJbXrRp+ii0uGNMnj2SXnzp6ld89uLFjiejVyYnj7jf60eKIV9Ro0TLJ9\nultQUBAtmjZk1e/ro11PTyxfjxlNhgwZnME/IerU8GXH9m1uC3deOYqYrK0/cUtbFyY9sz22hyQ8\nSElyzVNESgIewGXgb6C0iKQW6x6dhnaZ9EAmY8wS4DWsadb4Wg68ZI8wEZHiIpJORAoC540x32NN\nHVeyg67DGDMHGALEvEIlGRo4aZPb7628H/lzZGDI5C0xBs7kpLxPRWrUrut8SEJSKFm6zH0FzuQk\nV+7cdH++FzdiuA82MZQqXeahDpxgXfN8592hnDkd5332bpMpUyY6d4nXl4EoN0nMj7jeYj1zEKxB\nfDf7UUonRWQm1o2rx4GIuc0MwHwRSWOXfz0B+5qINYW7Q6x5u4tYN8fWA94SkRDgFtAV65mFP4pI\nxAeHQfd3eA/GvbfDPChHz17n6FnX11KTo2e7dE/S/XXuljxG3f/VU5EWzCSFHj1fSNL9JZZGjZsm\n6f66dOsRd6EklNwX+7hDogVPY0yM8xXGmLeBt11k+boo+36k9/Uivb+Efc3TGBMODLZfkU22X/d6\nqEabSin1MEkJwVOfMKSUUkolUPJYmaCUUuqREPGQhEedBk+llFLu9ejHTp22VUoppRJKR55KKaXc\nR1LGgiENnkoppdwqJQRPnbZVSimlEkhHnkoppdwqJYw8NXgqpZRyr0c/dmrwVEop5V4pYeSp1zyV\nUkqpBNKRp1JKKbcR0ScMKaWUUgmWEoKnTtsqpZRSCaQjT6WUUm6VEkaeGjyVUkq516MfO3XaViml\nlEooHXkqpZRyK522VUoppRIihXyrik7bKqWUUgmkI0+llFJuI0AKGHhq8FRKKeVOKeMJQzptq5RS\nSiWQjjyVUkq5VQoYeGrwVEop5V46bauUUkqpaHTkqZRSyn1Ep22VUkqpBBHA4Xj0o6dO2yqllFIJ\npMFTKaWUW4m45xX7PiSNiPiLyG4R2Sciw+z0QiKyVUSOiMgMEUllp6e2t4/Y+Y9HamuQnX5QRJrG\n5xg1eCqllHIrEXHLKw63gQbGmAqAD9BMRPyAkcCXxpiiwFWgp12+J3DVTv/SLoeIlAY6AmWAZsC3\nIuIR1841eCqllHroGMste9PLfhmgATDbTp8MtLHft7a3sfMbihWhWwPTjTG3jTHHgSOAb1z71+Cp\nlFLKfdw0ZWsPPLOLyLZIr7oTyLoAACAASURBVN5RdiXiISK7gAvASuAocM0YE2oXOQXktd/nBU4C\n2PnXgWyR013UiZGuto2HcoWys3xqz7gLqigKPTv+QXfhoXRxbr8H3YWHVkq4OT+5sx4M77b/D5eM\nMVViyjTGhAE+IpIZmAuUdNeO46IjT6WUUg81Y8w14HegOpBZRCIGhvmA0/b700B+ADs/E3A5crqL\nOjHS4KmUUsqN3LNYKK7Rq4jksEeciIg30Bj4CyuItrOLdQPm2+8X2NvY+WuMMcZO72ivxi0EFAP8\n4zpKnbZVSinlVkk0e54bmGyvjHUAM40xi0RkPzBdRIYDO4FJdvlJwBQROQJcwVphizFmn4jMBPYD\noUBfezo4Vho8lVJKPXSMMXuAii7Sj+FitawxJhhoH0NbI4ARCdm/Bk+llFJulRIWbmnwVEop5T4p\n5MHwumBIKaWUSiAdeSqllHIbN9/nmWxp8FRKKeVWKSB26rStUkoplVA68lRKKeVWOm2rlFJKJVAK\niJ06bauUUkollI48lVJKuY/otK1SSimVINatKg+6F4lPp22VUkqpBNKRp1JKKTeK++vEHgUaPJVS\nSrlVCoidOm2rlFJKJZSOPJVSSrmVTtsqpZRSCaFfSaaUUkopV3TkqZRSym30K8mUUkqp+5ASgqdO\n2yqllFIJpCNPpZRSbpUCBp4aPJVSSrmXTtsqpZRSKhodeSqllHKfFHKfpwZPpZRSbiP6YHillFIq\n4VJA7NRrnkoppVRCafBMJGFhYTSu7UuXDm2caT9M+JbqFUuRO3NqLl++5EyfM3MaDWpUpn6NSrRq\nUpd9e/e4bNMYQ7tWTbl54wYA348bS73qFanr58OEb8e4rBNT25cuXeTJZvWpV70iSxfNd5bv/uzT\nnDt7xrk9bMgANvzx+/2fiHuk9vJg/ZfPsPXrZ9k+rjNDOldz5q369Gm2jH2WLWOf5diU55n57hMA\nZEybitlDWznrdGlcylknf470LBzehp3jn2PH+OcokDODy/1+1rs2NcvmAaBehXxsGtORLWOfZfVn\n7SicO1OUsm1qFiFoyStUKpbTZVvj+zfk7197se3bzlHSh/eogf83nZj4RmNnWsf6JejX2se5Xebx\nbEx4rVF8TlW8vdS7J4Xy58K3Uvko6R99OIzihfNTw7cSNXwrsXzZEmfen3v30KBuTapWLEe1yhUI\nDg522fZzz7bn+LFjALRt1ZzqVStStWI5Xu33EmFhYdHKr/9jLXlzZnHu85MRHwJw8eJFGtevg2+l\n8ixcMM9ZvkO7Npw9c/fnbfDAt/jj9zX3fzIS4KXez/N4vseoWrGcy/wxX35B+tQOLl26+7u67o+1\nVK9akSo+ZWnaqJ7LesYYWjRtyI0bNwgODqZuzWr4VfGhik9Zhn8w1GWdsaNHUblCGapVrsATTRvx\nz99/A3Do4EFq+VWhWuUKbN2yGYDQ0FBaNmtMYGCgs363557lyOHD93MaEoVDxC2v5EyDZyL5ftxY\nipUoGSWtarUazJy3lHz5C0ZJL1DwcX5bsorfN+2g/1uDeKv//1y2uXrFUkqXLUeGjBk5sH8fv/z8\nA0tWb2T1hm2sWr6E48eORKsTU9vzZs+ga48XWLJ6I9+P+xqAFUsXUba8D7ly53HWf773/xg7+rP/\ndC4iux0SRrNBc6nWbxrV+k2jSZWC+JbIBUCjt+fg9/I0/F6extYD55i38SgAL7Ysz4F/LlOt3zSa\nDviNT3rVxsvT+tGd+EYTvpyznYp9plK7/wwuXg+Kts+sGdLgWzI3G/+0/kiP6VefHp8tx+/lacxY\ne5CBHas6y6b39qJvax/8D5yL8RimrPqL1u/Oj5KWMW0qfIrmxLfvr9wJDafM49lIk8qDro1LM37R\n3Q9D+05cJm/29OTPkf4+z2B0nbt0Y+6CJS7z+r7cn03+O9jkv4OmzVoA1h/fXj268tXYbwnYuZcl\nK9bg5eUVre5f+/cRFhZGocKFAZj8yww2B+zEf8ceLl26yNw5s1zus3rNWs59DnznXQBmz5xOzxd6\ns3bDFr4da33QW7J4IRUqVCR3nrs/b31e6seoz0fe/8lIgM5dujNv4VKXeadOnmT1qpXkL1DAmXbt\n2jVee6UvM+fMZ9uuP5ny60yXdZcvXUK5cuXJmDEjqVOnZvHy1WzZtovNATtZtWI5/lu3RKtT3qci\n6zcHsHX7bto89TRDBg8A4IeJ3/HpF6P5bf5ivvryCwC+/24cHTt1Jm3atM76vXr3YfSoT+/7XLib\niHteyZkGz0Rw5vQpVq9YSqcuPaKkl6vgQ/6Cj0crX7VadTJnzgJA5arVOHvmtMt2f5s1nWYtWgFw\n+NABKlX2JW3atHh6euJXsw5LFs6LViemtr28vAgKCuTOndt4eDgIDQ3l+3Fj+d+rb0Spn79AQa5e\nucKF8zEHk4T6NzjE6oOnA08PBwYTJT+Ddyrqls/Hws3WiMcA6b1TAZDO24urN4MJDQunZP6seHo4\nWLPzpLPdoNuh0fbXpmYRVmz/27ltjBXsADKmS83ZK/8684Z28eOLWdsJvhO9nQgb/zzDlZtRR2rh\nxuDlYf06pU3tSUhoOP2frsS4hbsJDQuPUnaJ/3Ha1y0e8wlKoFq165AlS9Z4l1+9agVly5ajXPkK\nAGTLlg0PD49o5WZM+5UnWj7p3M6YMSNgBd87d+4kaFGIl5cngYGB3L59Gw8PD0JDQ/l27Bj6v/FW\nlHIFChbkypUrnD/nvp+3mMR23ga89TrDPx4Z5RhnTv+VJ9u0dQbUnDldz0zMmP4rT7RqDVj3O6ZP\nb31QCgkJISQkxOV5q1uvvjMY+lbz48zpU4D9exoYSGBgIF5eXly7do2lixfR6bmuUerXrFWb31ev\nJjQ05p9b5V5JEjxF5B0R2Scie0Rkl4hUi6FcFREZE2nbS0SO23V2icg5ETkdaTtVAvsxXET6/9fj\nict7g95kyAcf43Ak/PROm/IjDRo1dZnnv2Uz5X0qAVCiVGm2bt7AlSuXCQwMZM3KZZw5dSrebbdt\n15HlSxbSoU0LXnljAD9NHE+7DlE/zUYoV6Ei/vaUkTs4HMKWsc/yz6+9WLPzHwIOno+S36p6Ydbu\nPsXNoDsAjF+4m5L5s3Jsak+2fduJN79bhzFQLF9mrv17m+nvtGDz2Gf56PmaOBzR/zBVL52HnYcv\nOLf/99Vq5g57kiM/P0+nBiX5fOZ2AHyK5CBfjgwsCziR4GO6FRTC8m0n2DL2Wc5d+Zcb/96maolc\nzg8Ake04fIEaZfImeB/3Y8K4b/Cr4sNLvXty9epVAI4cPoyI0KZlM2r5VeHLL1zPLGzZvImKlSpH\nSWvTshmF8+ciQ/oMtHmqnct6/lu3UL1qRZ56sgV/7d8HQPsOnVi8aAGtn2jKm28PdDl6ilDBpyJb\nNm/8L4f9nyxaMJ88efI4P1xEOHL4ENeuXqVZ4/rU8qvCr1N/dll/y+aNUc5bWFgY1atWpFC+x2jQ\nsBFVfV3++XOa/OMkGjdtBkDvPn357NOPebFXd94aMIiRH33ImwMGRfvb4nA4KFykKHv37L6fQ3Yr\na9QobnklZ4kePEWkOtASqGSMKQ80Ak66KmuM2WaMeSVSUi1gkTHGxxjjA4wHvozYNsbcSez+J9TK\nZYvJniMHFewglxAb163l1yk/8c6wES7zr127QvoM1jW94iVK0ffVN+nY9gk6Pd2KMuXK43Axeoip\n7YyZMjF15nyWr91MuQoVWblsMS1bP8Ubr7xEr64d2eZ/d2ope44cnD93JqamEyw83OD38jSKdv2B\nKsVzUbpg1E//z9Qrzsw/Djq3G1cqyJ5jFyn83CSq9ZvGly/VJYN3KjwdDmqWycPASRuo9ep0CuXO\nRJdGpe7dHbmypuNSpOncl9v40HboAop2/YEpK/czsndtRGDkC7UZ8P36+z6uUbN34PfyNAZO3MB7\nXavz4ZQtdG9ahqmDmjMg0tTwhWuB5M6a7r73E1+9evdhz1+H2eS/g1y5cjN4wJuANXLcvGkjE3+a\nyoo161i4YB5r16yOVv/cubNkz54jStq8Rcs4fOI0t+/cdnltskLFSuw/dJzNATt58X/9eLb9UwBk\nypSJOfMWsW6TPxUqVmLp4oW0eaod/V7qzXPPtndezwPIkSMnZ8+edeepiLfAwEA+//Rjhgz9IFpe\naGgou3buYM68RcxbtIyRHw3n8KFD0cpdvXKFDBnuXnv38PBgc8BODh47ybZtAezb92eM+5/+61R2\n7thO/9etEXn+AgVYtvJ31qzbhLd3Wk6fPk2JkqXo1aMrXTt3jLL/HDlzRrl+/CA5xD2v5CwpRp65\ngUvGmNsAxphLxpgzIlJVRDaJyG4R8ReRDCJST0QWRarbDHB9UcImIt3s+rtE5FsRcdjpT4jIDrv9\nFZGqlBORP0TkmIj0dffB+m/dzIqli6larjh9enZhw7q19O3dPc56+//cyxuv9OGnX2eTNWs2l2U8\nPTwJD787Bdipaw9W/LGFeUtXkylzFooULXZfbX/56Ue8+sZA5s6ZQTW/GowZN4nPP/nQmX87OJg0\n3t5xHkNCXf/3Dn/sOUWTynevAWfLmIYqxR9jqf8JZ1qXxqWYv8m6/nns7HVOnL9BifxZOH3pFnuO\nXeLEuRuEhRsWbD6GT9HoU2lBt0NJncr6YJE9ozflCudwjnZnrzuEX6ncZPBORemC2Vgx8mkO/Ngd\n35K5mP1eyxgXDcWmQuEcCHDo1FWeqlWU5z5eSuHcmSiSx1qYlCaVZ6zTwu6S87HH8PDwwOFw0P35\nXmzfFgBA3rx5qVGrNtmzZydt2rQ0bdqcXbt2Rqvv7e3tciFRmjRpeKLlkyxetCBaXsaMGZ3TlE2b\ntSAkJCTKghuAkR8P560Bg5k1YxrVa9Tku4k/8fHwYc784NvBpEmT5j8d+/06duwoJ04cp3pVH0oX\nL8TpU6eo5VeZ8+fOkTdfPho2bkK6dOnInj07NWvXZu/e6CM9T8+ov6cRMmfOTJ269Vi1fJnLff++\nehWffvIRM+bMJ3Xq1NHyhw0dwnvvf8i4b8bQrUdPhn80ko9H3A3ywcHBeCfC76lyLSmC5wogv4gc\nsoNbXXu6dQbwqjGmAtZoNPpKD6gPrI2pYREpC7QFatgjU0+go4jkAsYBbe32O0aqVhxoDPgBH4iI\ny+GaiPQWkW0isi3yyti4vDN0ODv2HyNg7yHGT5pCrTr1+GbCT7HWOXXyH3p2eYax3/1IkaIxXwsr\nUqw4f5+4Ow146eIFZ/0lC+fRtl3HaHXiavvY0cOcPXOaGrXrEhQYiDgcIEJw0N0/mseOHKZkqTJx\nHXq8ZM/oTaZ01mx7mlQeNKyYn4Onrjrz29YqylL/E9wOubuS8+TFm9TzyQ9AzszeFM+bhePnrrPt\n8HkypUtF9ozWH4x6FfJx4J8r0fZ58OQViuTJDMDVW8FkTJuKonmt7QYVC3Dw5BVuBN4h/7PfU7LH\nT5Ts8RP+B87R7oNF7Ig03Rtf73X144MpW/DydOBhf3wODzekTW0tyimWNzP7/r6c4HYT6lyk0dvC\nBfMoXcb6f9iwcVP27/uTwMBAQkND2bB+HSVLRR+xlyhZkmNHrUVot27dcrYXGhrK8mVLKH7PgjiA\n8+fOYYx1DXtbgD/h4eFky3b3A9uRI4c5c/oUtevWIygoCIfDgYgQFOnn7cjhQ5QuU9YNZyDhypYt\nx4lT59l/6Dj7Dx0nb758bNiyncdy5eKJlq3ZvHEjoaGhBAYGEuDvT4mS0c9bseIlnCuUL168yLVr\n1wAICgpizepVLs/b7l07eaVvH2bOme/yWur6dX+QO3duihYrRlBgIA6HA4fDEWXF7YM8b/dKCdO2\nif6QBGPMLRGpDNTGCoYzgBHAWWNMgF3mBkR9mLCI5AWuGGMCozV6VyOgKrDNruuNNSUcBPxujPnb\nbj/yX9RF9nTvBRG5AuQAoq1OMMZMACYAVKhY2dybfz8mjv+ab8eM4sL5czSsWYWGjZvxxdjxfPnp\nR1y9coVBb1gz1h6enixfG/0aY8Mmzdm0YR2FChcFoGfXjly9chkvTy8+/vwrMmW2AsLkHyYA0O35\n3nG2/cmHQxn4rvWpv227DvTo3J6vR3/GW4OsJfUhISEcP36UChWjXvu6X7mypuX7N5rg4bCWos9Z\nfzjKKLN9neJ8Pmt7lDqfTAtgwuuNCfi2E4Lwzo8buXzD+mM7aNIGlnzcFhHYefgCPyyLPiW2LOAE\nPZuX5afl+wgLN/Qds5pp77QgPNxw7dZtXhy9KtY+586ajm9fbUjbodZIa/LbTaldPh/ZM6bhyM/P\n8+HULUxesR+wrtfuOHzBuQhpz7FLBHzbiT+PX2LvcetDWN3y+VgW6Zj/qx5dOrF+/R9cvnSJEkUK\nMHjIULr16Mm7gwewZ89uRIQCBQsy5uvxAGTJkoV+r/Snbs1qiAhNmjWnWfMnorXbtFkL1q/7g/oN\nGxH47790aNeG27dvEx4eTp269ej5wosATPrearfnC32YN3cOEyeMx9PTkzTe3vw45dcov9cfDB3C\ne8OGA9D+mY50fOYpRn3+KUPeex+wft6OHT1KpcpV3HZ+YtK9SyfWr1vL5UuXKF44P++8+z7devSM\nsXzJUqVo3KQp1SpXsEbzPXpSxkWwatq8BevXraVI0aKcP3eW3j27ExYWRnh4OE+1a0/zJ1oC8OGw\n96hUqQpPtHqSdwa+za1/b9Gl0zMA5M9fgJm/WSu6jTF8+vEIJv8yHYAePXvTs/tzhIaGMnrstwCc\nP38eb29vHsuVy52n6L4l87jnFhLxKTHJdijSDugLpDLG1Lwnrx7wpjGmpYj0BDIaY76MlP8+cMsY\n87m9/RqQ1Rjz7j3ttAXaGGO63ZM+HGsKebS9fQBoZIyJdaVNhYqVjatgltTOnzvLK32eZ8a8WGey\n3WrJwvns3b2TAUPeT3DdQs+Od3+H7tPqz9rx1PsLuP7vg71MnsrTg5WfPk2DN2cRFu76d+/i3H5J\n3CvXgoKCaNG0Iat+X+9yNW5iWDB/Lrt37uTd96Nfc4yP5DBaOXf2LC88342FS1fEXdhNvv7qSzJk\nzBhr8I9J7epV2bF9m9tOXKaCpUytwZPd0taSPtW2G2MS/5PUfUiKBUMlRCTyxTgf4C8gt4hUtctk\nEJF7R8FxXu8EVgHPiEh2u51sIlIA2ATUF5GCdnr81/EnY4/lyk3nbj2dD0lICmFhofTpl+gLlBPd\nwInryZ/D9QMUklL+nOkZ8uPGGANncuLt7c077w7lzGnXt04lhrDQUF7u/3qS7S8x5Mqdm+49e3Ej\nCX9PM2XOTOcu3eIumAQE+/m2bvgv1v2I5BeR30Vkv303x6t2+vv33JXRIlKdQSJyREQOikjTSOnN\n7LQjIjIwPseZFM+2TQ+MFZHMQChwBOgN/Gine2NNszofu2JfhyxqjDkQW8PGmL0iMgxYZS8UCgH6\nGGMCROQlYL5YH0XPAM0T4diS3JNtXd8ekFhatXk6SfeXWO69HeZBOXrmOkfPXH/Q3Yi3Ro1d3zaV\nWNo+3T5J95dYnm73TJLur0u3HnEXSkJJtFI2FHjDGLNDRDIA20VkpZ33ZcQMZQQRKY21/qUMkAcr\nbkQsBPkGay3MKSBARBYYY/bHtvOkuOa5HajhIusS1qKdyNYCa0WkFrDVRVvvu0j7FfjVRfpiYPE9\naUPu2Y5+5V4ppVSyZ4w5C5y1398Ukb+A2G6gbg1Mt+/8OC4iRwBfO++IMeYYgIhMt8vGGjyT5ROG\njDEbjDF9HnQ/lFJKJZCbVtra16+zR9z1YL96u96lPA5U5O6gq59YD+X5QUSy2Gl5ifqMgVN2Wkzp\nsdKvJFNKKeVWbly3dSmuBUMikh6YA/Q3xtwQkXHAh1hP9vwQ+AJ43m09smnwVEop9VASES+swPmL\nMeY3AGPM+Uj53wMRD945DeSPVD2fnUYs6TFKltO2SimlHk5C0nwlmb0YdBLwlzFmVKT03JGKtQUi\nbv5egPUQndQiUggoBvgDAUAxESlkP8Cno102VjryVEop5VZJdLttTaALsFdEdtlpg4FnRcQHa9r2\nBPAigDFmn4jMxFoIFAr0NcaEWf2VfsBywAP4wRizL66da/BUSin10DHGbACXN4O6/nJbq84IrCfc\n3Zu+JLZ6rmjwVEop5VbJ4UlPiU2Dp1JKKbexvs/zQfci8emCIaWUUiqBdOSplFLKreJaKfsoiDF4\nikjG2CpGfI2YUkopFdmjHzpjH3nuw1rqG/k8RGwboEAi9ksppZRKtmIMnsaY/DHlKaWUUjFJCatt\n47VgSEQ6ishg+30+EamcuN1SSin1MLKeMOSeV3IWZ/AUka+B+lhPcgAIBMYnZqeUUkqp5Cw+q21r\nGGMqichOAGPMFfv5f0oppVRUd79O7JEWn+AZIiIOrEVCiEg2IDxRe6WUUuqhlQJiZ7yueX6D9ZUv\nOURkGLABGJmovVJKKaWSsThHnsaYn0VkO9DITmpvjPkztjpKKaVSLp22vcsDCMGautVH+imllHIp\nYrXtoy4+q23fAaYBebC+YftXERmU2B1TSimlkqv4jDy7AhWNMYEAIjIC2Al8nJgdU0op9XDSaVvL\n2XvKedppSimlVDSPfuiM/cHwX2Jd47wC7BOR5fZ2EyAgabqnlFJKJT+xjTwjVtTuAxZHSt+SeN1R\nSin1MBNJ4V9JZoyZlJQdUUop9WhIAbEz7mueIlIEGAGUBtJEpBtjiidiv5RSSqlkKz73bP4E/Ih1\nDbg5MBOYkYh9Ukop9RAT+/m2//WVnMUneKY1xiwHMMYcNcYMwQqiSimlVDQi7nklZ/G5VeW2/WD4\noyLSBzgNZEjcbimllFLJV3yC52tAOuAVrGufmYDnE7NTSimlHk6CpOzVthGMMVvttze5+4XYSiml\nVHQPwZSrO8T2kIS52N/h6Yox5qlE6ZFSSimVzMU28vw6yXqRzHk6hMzpUj3objx0ri545UF34aGU\npWq/B92Fh9blrWMfdBcUKfzZtsaY1UnZEaWUUo+GlPC9lSnhGJVSSim3iu+XYSullFJxElL4tO29\nRCS1MeZ2YnZGKaXUw8/x6MfOuKdtRcRXRPYCh+3tCiKiV+WVUkq55BD3vJKz+FzzHAO0BC4DGGN2\nA/UTs1NKKaVUchafaVuHMebve+awwxKpP0oppR5i1nNpk/mw0Q3iEzxPiogvYETEA3gZOJS43VJK\nKfWwSu5Tru4Qn2nbl4DXgQLAecDPTlNKKaVSpDiDpzHmgjGmozEmu/3qaIy5lBSdU0op9fBJiq8k\nE5H8IvK7iOwXkX0i8qqdnlVEVorIYfvfLHa6iMgYETkiIntEpFKktrrZ5Q+LSLf4HGOc07Yi8j0u\nnnFrjOkdnx0opZRKOQSS6ltVQoE3jDE7RCQDsF1EVgLdgdXGmE9EZCAwEBiA9T3UxexXNWAcUE1E\nsgJDgSpYsW67iCwwxlyNbefxmbZdBay2XxuBnIDe76mUUuqBMcacNcbssN/fBP4C8gKtgcl2sclA\nG/t9a+BnY9kCZBaR3EBTYKUx5oodMFcCzeLaf3y+kmxG5G0RmQJsiM/BKaWUSnmS+rmvIvI4UBHY\nCjxmjDlrZ50DHrPf5wVORqp2yk6LKT1W9/N4vkKROqOUUkpF4cZZ2+wisi3S9gRjzISo+5L0wByg\nvzHmRuTbZIwxRkRi/GrN/yI+1zyvcveapwO4gjWHrJRSSiWmS8aYKjFliogXVuD8xRjzm518XkRy\nG2PO2tOyF+z000D+SNXz2WmngXr3pK+Nq2Oxjq7FCuEVgBz2K4sxprAxZmZcDSullEp5RASHm15x\n7EeAScBfxphRkbIWABErZrsB8yOld7VX3foB1+3p3eVAExHJYq/MbWKnxSrWkac95F1ijCkbV0NK\nKaUUuHXaNjY1gS7AXhHZZacNBj4BZopIT+Bv4Bk7bwnQAjgCBAI9AIwxV0TkQyDALveBMeZKXDuP\nzzXPXSJS0RizM54HpJRSSiUqY8wGrDtjXGnoorwB+sbQ1g/ADwnZf4zBU0Q8jTGhWCuYAkTkKPCv\n3VljjKkUU12llFIpV0p4PF9sI09/oBLwZBL1RSml1EMuCR+S8EDFFjwFwBhzNIn6opRSSj0UYgue\nOUTk9Zgy71ndpJRSSgFJtmDogYoteHoA6Yn5gqxSSikVleg1z7PGmA+SrCdKKaXUQyLOa55KKaVU\nQkgKCB+xBc9o98kopZRSsbFW2z7oXiS+GB/PF58nLCillFIp0f18q4pSSikVo5Qw8tTgqZRSyq0k\nBdyrktTfWaqUUko99HTkqZRSym1SyoIhDZ5KKaXcR1LGE4Z02jaRjRn9JZUqlKGyT1m6PvcswcHB\nALzwfHdKFitEtco+VKvsw+5du1zW37VzJ31e6AnAwQMHqFurOpnSpebLUZ/HuM+1v6+hetVKVPYp\nS68e3QgNDQVg7m9zqFShDA3r1eby5csAHDt6lOc6dXDWvXPnDo3q13HWeVC+HvMVlX3KUqlCGcZ+\nNdqZPmjAW1QoW5KqFcvzTLu2XLt2zWX9s2fP8lTrlgCsXrWSGr6VqeJTjhq+lVn7+xqXda5cucIT\nzRpTtlQxnmjWmKtXrwLJ67w5HMLmaQOY81UfZ1qfDnX4c/5QgnZ+TbbM6ZzpmTN4M+OLF/CfMYj1\nU96kdJHczrwDi4cRMHMwW6YPZMMvb8e4v36d6tGppS8A5YrnZe3kNwiYOZjZo18kQ7o0AFQpU5At\n0weyZfpAts4YyJP1y7ts68cR3dg99122zRrM+KGd8fS0/vy0aejD9tnvsGpSf7JmsvpfKF92pnzS\nw1nXy9ODlZP64+Hhvj9ZfXo/T8F8j1GlYrko6YMHvkXFcqXwrVyBju2fcv6MhYSE8ELP7lStVJ5K\n5Uvz2acfu2zXGEPzpg25ceMGp06epHmTBlSuUIYqPmX5ZuxXLutcv36ddm2fpFoVH6r4lOXnyT8C\ncOjgQWr6VcG3cgW2kTDjFQAAIABJREFUbtkM/2/vvuOjKNoAjv+eFHroLSBFaui9gyK9N5HeO4K+\nNhAVFBWkI9JEQKSJgAWkF5HeEzpKb9IFpCYEEub9YzfHhUsCCQcE8nz53Cd3szuze8PePTuzcztA\nSEgItWtUJTAw0JG/bavmHDl8+HGrREWDBs8n6MyZM4wfN5qNW/wJ2LWP0NBQfp4z27H8q8HD2Bqw\ni60BuyhUuHCEZQwd8hVv9nwbgBQpUzLi69G8894HkW7z3r17dOrQluk/ziZg1z4yZ8nCzOnTAPh2\n3Bg2bN5Op85dmfPTLAD6f9aX/p8PcOSPFy8er1WqzM9z5zz2+4+p/fv28cOUSazftI1tAbtZumQR\nR48cAaBylaoE7NrH9p17yJkzF8OGRPwFNnrUSNp37AxAqlSp+WX+Qvx37WXSlGl0aNc6wjzDhw6m\nYqXK7Pv7MBUrVWb40MFA7Kq3ni1e4+DxC+HSNu86Rq1uYzh59nK49N4dq7P74GlKNh1Ex34zGN6r\ncbjlNbp8Q+lmgynfcmiE2/L09KBN/TLMWeoPwLeftqDv6N8p0eQrFqzezbttrZ+C7z96lnIth1K6\n2WDq9xjPmL7NIwxys5dup1DDLyn+xlckTOBN+4ZlAeje7FXKtxrK5F830rRmcQD696hD//GLHHnv\nhoSyeutB3qjmvpkQW7Vux/yFS13SK1Wuyvade9kWsJscOXMy3A6Sv/36M3eCg9m+Yw8btvgzZfJE\nTp444ZJ/+dIlFChQkKRJk+Lp5cVXQ4YTsHs/q9dvZuKE8fz9918ueSZOGIdfnjxs9d/F0pWr+fjD\nD7hz5w7fT/6OYSNGMe/3xXzz9QgAJn33Lc1atCRRokSO/J26dOPrkRH/Pz4LHiJuecRmGjyfsJCQ\nEIKCgqy/gYH4ZsjwyHlv3LjBvr17KFioEABp06aleIkSeHt7R5rn8uXLxIsXj5y5cgFQqUpV5s/7\nFQAPDw+Cg4MJDAzE29ubDRvWky5denLkzBmujLr1GjDnpx+j+1bd5sCBvylRohSJEiXCy8uLCq+8\nyvz5vwFQpWo1vLysqw0lS5XmzOnTEZYxf96vVKteA4DCRYqQwa73vPnycTsoiODgYJc8ixb+TqvW\nbQFo1botCxfMB2JPvWVMm5wa5fPxw7xN4dJ3HzzNqXOuP8v2y5aetdsPAXDoxAWyZEhJ2pQ+j7y9\niiVysevAP4SG3gMgR+a0bAiwTmL+3HKABpWtE76g23cd68SP540157Cr5RvuBw3/fSfJmDYFYJ3w\nxff2IlGCeNwNCaVckexcuHSdo6f+DZd/4Zo9NK1V4pH3/2HKV3iFlClSuqS7HGNnzgDWCNJbt245\nPtPxvOPhkzSpS/7Zs2dRp259AHx9fSlSxAr4Pj4+5PbLw1m7vHBEuHnjBsYYbt28SYoUKfHy8sLb\n25vAwEDHsXf16lWWLF5Ey1ZtwmUvV74Cq1eteuY9RnD/mqc7HrGZBs8nKGPGjLzz7gfkypaZlzP5\nkjRpMqpUreZY3v/TTyhRpCC93n83wi/zHQH+5M2XP1rbTJ06NSEhIQT4W62Feb/+wul//gGg14cf\nUbt6FZYsXkiTZs0ZPPBLPvqkn0sZ+fLnJ8B/e7S260758uVn48b1XL58mcDAQJYtXeJ4D86mT51C\n9Ro1XdJPHD9OiuQpiB8/vsuyeb/9SuEiRSNcdvHCBXx9ra7N9OnTc/GC1cKLLfU2rNfrfPLNfO7d\nizg4PWjvoTPUr2SdeBXPl4XMvinJmC45YHUtLhzfk40/9qZDo3IR5i9TOBs7/75f738fO0fdilaX\nbKOqRXkpXQrHshL5sxDwyyf4//wxbw+c7QimEfHy8qB57ZKs3GQF02FTVrJ4wlvUeiU/c5f506dz\nDQZNWuaSb/+RsxTLl/mR3ru7TJ/6g+MkrGGjxiROnJjsWTLglyML/3v3fVKmdA2+WzZvpEjRYi7p\nJ0+cYPfunZQoWcplWbfuPTl48ADZs2akZLGCDBsxCg8PD7p268HwoYPo0qkdH3z4EYO/+pJeH36E\nh0f4r24PDw+yZc/B3j273fTO1cPEquApIp+IyH4R2SMiu0TE9SiLfpkVRaSsO/Yvuv777z8WLfyd\nvw8f59ips9wKvMVPP84E4IuBg9i97wAbtmznvytXGDFsiEv+c+fOkSZ1mmhtU0SYPnM2vT94l/Jl\nSuLj44OnpydgdXlu2hbAr/MXsmjB71SvUYvDhw7RvGlj3uza2XENxdPTE+948bhx48Zj1kDM+OXJ\nw/sffEjdmtWoV7sGhQoVdryHMEMGDcTTy4tmLVq65D937hyp07jW21/799P34w8ZO/67h+6DiDh+\nqxYb6q1mhfxcvHIjXDB7mOE/rCSZTyK2zO5D92avsvvgaUdQq9z+a8q2GEKDnuPp2rQC5Ypmd8mf\nPnUyLv130/G6a/8f6dKkAht/7E2SRPG5czfUsWz7vpMUazyQ8q2G0qtDNeLHi3ws4jcfNWXjjiNs\n3GlNFfzn1gOUazmUxu98R52KBVm+YT85s6Rl1rCOjOvXnIQJrJ6We/cMd++GkiSR64nPkzB08EC8\nvLxo1tw6xvy3b8PD05MjJ86w/+AxRo8ayfFjx1zy/XflCj4+4Vv4N2/epEWzxgwd/jVJI2it/rFy\nOQUKFuLoiTNs3raT9955i+vXr5Mpc2aWrVzN6nWbSJQwEWfPnCG3Xx46tm9Dm5bNOHzokKOMNGnT\ncu7sWTfXQsyIuOcRm8Wa4CkiZYA6QFFjTEGgCvDo3xQRl+kFVASeSfD8c9UfZM36MmnSpMHb25sG\nDRqxZbPV5ebr64uIED9+fNq0a4//9m0u+RMmTOgYYBQdpcuUYdWa9WzYvI3yFV4hh92FGyYwMJAZ\n06fS7c0eDPjiMyZPmUbZcuWZPet+l+Od4GASJEgQ7W27S7sOHdm0LYA/Vq8jeYoU5Mx5/z3MmDaV\nJYsXMXX6jxH+GDuiejt9+jRN32jI5CnTyZbdNVAApE2XjnPnzgH2iUvatOGWP8t6K1M4G3VeLcCB\nxZ8zfXB7KpbIxZQBbaLMc+PWbbr2n0npZoPp2G86qVMk4fgZ67ro2X+vAfDvfzdZ8OceSuTL6pL/\ndvCdcEHw0IkL1H1zHOVaDmXusgCOn/7XJc/B4xe4GRhMvhwRX574uEtN0qRIQu8Rv7ksS5jAm9Z1\nSzFh7jr6dqtNp34z2LTrGM1q3u+qjeftxe07d6N83+4wY/pUli5ZzJRpMx3H2NzZs6harTre3t6k\nTZuW0mXLsmOHv0teLy8v7t273/K+e/cuLZo2pmmzFtRv0Cji7U2bSv0GjRARsufIQZaXX+bQwQPh\n1vn8s7582v9Lvh03mnbtOzLgqyF8NfD+xFfBt2+TIGFCd7z9xyR4uOkRm8Wa4An4ApeMMcEAxphL\nxpizInJCRIaKyF4R2SYiOQBEJKuI/Gm3UleJSGY7faqITBCRrcBcoBvwrt2SrSAib4jIPhHZLSLr\nnuQbypQpM9u2bSEwMBBjDKv/XEVuvzwAji9pYwwLfp8fYfesn18ejh49Eu3tXrx4EYDg4GBGDBtC\n5y7dwi3/esQw3uz5Nt7e3gQFBSEieHh4OFpQly9fJlXq1FFeW33Swt7DqVOn+H3+bzRt3gKAFcuX\nMXLEUH6ZtyDcgAlnOXPl4uTJE47XV69epVG92nw5cDBly0XcRQlQu049Zs6wBlfNnDHNcd0qzLOs\nt0/HLCBHjX741f6MNn1+YM32Q3ToOz3KPMmSJMTby2qxt29Ylg07jnDj1m0SJYjnaL0lShCPKmX8\n2H/UtcVy4PgFsme634JPkyIJYLXK+3SuzqRfNgCQJUMqxwChzL4pyP1yepfBSwDtGpahatk8tPlo\naoTXRd9tU4XxP60lJOQeCRN4YzDcu3ePRAniAZAyWWIuX71JSEjkXcLusGL5MkaNGMbcX38Pd4y9\nlDkza9esBuDWrVts37qVXLn9XPLnzJXb0SI1xtC9aydy+/nx9jvvRbrNTJkysWb1KgAuXLjA4UMH\nyfpyNsfy9evWkt7Xlxw5cxIYGIiHhwceHh4EOY24PXz4ULQv86iYi02/81wBfCoih4A/gDnGmLX2\nsmvGmAIi0gYYhdVCHQNMM8ZME5EOwGiggb3+S0BZY0yoiPQHbhpjhgOIyF6gujHmjIgkj2xnRKQL\n0AUgU+aYXWcpWaoUDRs1pkzJonh5eVGoUBE6du4CQPs2Lbn0778YDAULFmbM+Aku+XP7+XH92jVu\n3LiBj48P58+fp1zp4ty4fh0PDw/Gjh7Fzj1/kTRpUhrUrcX47yaTIUMGvh4xjKVLFnHv3j06d+lO\nxdcqOco8e/Ys/tu38Um/zwDo3uMtypcpQbJkyZn7qzVAZu2a1dSoWTtG79ldmjd5nStXLuPt5c2o\n0eNIntz6r3r3fz0JDg6mTo2qgDWg48G6S5w4MdmyZefokSNkz5GDCePHcvToEQYN+IJBA6wz9YVL\nV5A2bVq6d+lEpy7dKFa8OB/07kOr5k2Y9sP3ZM6chZk/zXWUGZvr7c3mr/Je2yqkS5WU7XM/ZtmG\n/bz5xSz8sqVn0hetMcbw99FzdPvcaiGnTeXDnJHWSGQvT0/mLPVn5aa/XcpdsXE/3w9o63jdpEZx\nujZ9BYDf/9zF9N+3AFC2SDY+aF+NuyGh3Ltn+N9Xc7h89RYA88Z0580vZnHu32uM+bgZp85dYc20\n9x1lDJpoXdv0TZOM4vmz8NVEa/Trtz+tZcPM3ly7EUiT9yYB8GqJnCzbsN9t9da2dQvWr1vD5UuX\nyJktE3379adt+468/85bBN8Jpm4ta3xCyZKlGD1uAl279aBb5w4UL5wfYwyt2rSjQAHXn+XUqFmL\n9evWkD1HDjZv2shPP84gX/4ClC5RBID+XwykRs1aTJ5oHbedunSjz8f96NKpPSWKFsQYw5cDB5M6\ndWrACsBDBg1k+o/WSP0OHbvQoV0rQkJC+GbMeMAKuAkTJiR9+vRuq5+YEmJ/l6s7SGQj454FEfEE\nKgCvAV2BPkB/oJIx5piIeAPnjTGpROQS4GuMuWunnzPGpBaRqcBqY8w0u8z+hA+eE4DsWK3S34wx\nrqfIDyhWrLjZuNW1e+ZpGD3qa3x8fGjfsdNT22bTNxoxYOBgx4jd59Hv8+exc0cA/b8Y8PCV3cRd\n9ZaiRE837dHjmzOiMx9/M99l5OuzMHt4J/qOXsCRUxcjXefy1jFPcY8idu7cOTp3aMuipSue2jbH\nfGNdS23bvmO085YvU4IdAf5uC3dZ8hQ0H01Z4Jayupd9OcAYU9wthblZbOq2xRgTaoxZY4z5DOgJ\nvB62yHm1RyjqVhTb6Ab0BTIBASKSKqb7+zR06dY9wpGhT8qdO3eoV6/Bcx04Aeo3aEiWLFmf2vZe\nlHp7UN/Rv5M+tesAl6fN28uTBWv2RBk4YwtfX1/ad+zE9evXn9o2kyVPTsvWbR++onKbWBM8RSS3\niDj/cK4wcNJ+3tTp72b7+Sagmf28JbA+kqJvAI6hbyKS3Riz1RjzKfAvVhCNtRIkSECLVhH/qP9J\niBcvHi1bRz0Q5XnxNFvrL1K9OTt88iIbdxx91rvB3ZBQZi1yHVQXW73euEmEo2qflDZt2zt+mxob\nxIWbJMSe2oYkwBj7OmQIcATrmmMdIIWI7AGCgeb2+m8BP4hIL6wg2N61SAAWAr+ISH07z7t2kBZg\nFaA/jFJKKTeJK9c8Y03wNMYEEMFPSuxh4sOMMR8+sP5JoNKD6xtj2j3w+hDgfFU/shaqUkop9Uhi\nTfBUSin1YojtXa7uEOuDpzEm67PeB6WUUo8uDsTO2DNgSCmllHpexPqWp1JKqeeHEDdaZRo8lVJK\nuY8Q4T2nXzRx4QRBKaWUcitteSqllHKrF7/dqcFTKaWUGwlx46cq2m2rlFJKRZO2PJVSSrnVi9/u\n1OCplFLKzeJAr6122yqllHo+icgUEbkoIvuc0vqLyBkR2WU/ajkt+0hEjojIQRGp7pRew047IiJ9\nHmXb2vJUSinlRvI0f+c5FRgLTH8g/WtjzPBweyWSF2say3xABuAPEQmbgHccUBU4DWwXkQXGmL+i\n2rAGT6WUUm7zNO8wZIxZJyJZH3H1+sBsY0wwcFxEjgAl7WVHjDHHAERktr1ulMFTu22VUkq5lYi4\n5QGkFhF/p0eXR9yFniKyx+7WTWGnZQT+cVrntJ0WWXqUNHgqpZSKrS4ZY4o7PSY+Qp5vgexAYeAc\nMOJJ7Jh22yqllHKrZznY1hhzwbEfIpOARfbLM0Amp1VfstOIIj1S2vJUSinlPuLWbtvob17E1+ll\nQyBsJO4CoJmIxBeRl4GcwDZgO5BTRF4WkXhYg4oWPGw72vJUSin1XBKRn4CKWNdGTwOfARVFpDBg\ngBNAVwBjzH4RmYs1ECgE6GGMCbXL6QksBzyBKcaY/Q/btgZPpZRSbvOUR9s2jyD5+yjWHwgMjCB9\nCbAkOtvW4KmUUsqtdD5PpZRSSrnQlqdSSim3evHbnRo8lVJKuVkc6LXVblullFIqurTlqZRSym2s\n0bYvftNTg6dSSim30m5bpZRSSrnQlqdSSik3EkS7bZVSSqno0W5bpZRSSrnQlqdSSim30dG2Siml\nVHRJ3Oi21eD5CO4ZuH0n9FnvxnPH0yMOfIKegHObvnnWu/DcSlVv1LPehedO8JELD19JudDgqZRS\nyq205amUUkpFU1z4qYqOtlVKKaWiSVueSiml3EaAuDDcQYOnUkopt9JuW6WUUkq50JanUkopt9LR\ntkoppVQ0abetUkoppVxoy1MppZTb6GhbpZRSKtrixnye2m2rlFJKRZO2PJVSSrmPzqqilFJKRV8c\niJ3abauUUkpFl7Y8lVJKuY012vbFb3tq8FRKKeVWL37o1G5bpZRSKtq05amUUsq94kDTU4OnUkop\nt9KbJCillFLKhbY8lVJKuVUcGGyrwVMppZR7xYHYqd22Simlnk8iMkVELorIPqe0lCKyUkQO239T\n2OkiIqNF5IiI7BGRok552trrHxaRto+ybQ2eSiml3Evc9Hi4qUCNB9L6AKuMMTmBVfZrgJpATvvR\nBfgWrGALfAaUAkoCn4UF3Kho8FRKKeU2Vtxzz7+HMcasA648kFwfmGY/nwY0cEqfbixbgOQi4gtU\nB1YaY64YY/4DVuIakF3oNU+llFKxVWoR8Xd6PdEYM/EhedIZY87Zz88D6eznGYF/nNY7badFlh4l\nDZ5KKaXcx71Tkl0yxhSPaWZjjBER47a9caLdtkoppdzq6V3yjNAFuzsW++9FO/0MkMlpvZfstMjS\no6TBUyml1ItkARA2YrYt8LtTeht71G1p4JrdvbscqCYiKeyBQtXstChpt61SSin3eko/9BSRn4CK\nWNdGT2ONmh0MzBWRjsBJoIm9+hKgFnAECATaAxhjrojIl8B2e70vjDEPDkJyoS1PNzt9+h/q1qxM\n6WIFKFO8IBPGjXYsGzzwc/LmyEyF0sWoULoYK5YtcSzbt3cP1V4rR5niBSlbojC3b9+OsPy2LZtw\n4vgxAOrUqESJwnkd5f178aLL+nNnz3Isr1C6GCmTeLN39y6Cg4NpXL8WZYoXYvLEbx3rv9OzG7t3\n7nC8njhhHDOn/fDY9fIo3uzakWyZ01OqWMFw6Xt276LSK2UpV6oor5Yrif/2bQBcu3aNJq/Xo2zJ\nIpQsWoCZ0yPez6CgIGpWfY3Q0FBH2vXr1/HLnpn333krwjx9P+pNsUJ5KVOiMC2aNOLq1asAbNm0\nkTIlCvNquZIcOXIYgKtXr1K/TnXu3bvnyF+vVjX++++/mFfGIzp9+h/q1qhM6aIFKFMs/PEWZuw3\nI0mRyIvLly4BcOjgAapVLEe65IkYM2pEpGUbY6hXswrXr18HoKBfdsqWKEyFUsV4rVypKPdrh/92\nUvvE5/d5vwJw+NBBKpYtSbmSRdi2dTMAISEhNKhdjcDAQEe+Dm1acNSuV3eI7+3J+m+as3V8KwK+\na0PfVmUcyyoWzsSmsS3YMq4lq0Y0IZtvMgDK5c/IprEtuLH4fzQsnzNceTcX/48t41qyZVxLfu5f\nL9LtDuv6KuXyZ4xyO2EalMtB0LJ3KZozXURF8VbDIgR81wb/Ca2Z1qcm8b09Afihdw22fduKz9uV\nc6z7YfOS1C2T3fG6ZsmX6de6jEuZT5a7xto+0mjb5sYYX2OMtzHmJWPM98aYy8aYysaYnMaYKmGB\n0B5l28MYk90YU8AY4+9UzhRjTA778UhfeBo83czL04sBXw1jS8BeVqzeyOSJ33Lg778cy7v3/B/r\ntwSwfksA1WrUAqwvka4d2zLim/Fs9t/DomWr8Pb2din777/2ExoaStaXsznSJk6Z7igvTdq0Lnma\nNGvhWD5h8lSyZH2ZAoUKs+qPFZQuW46N23Yy96eZAOzds5vQ0FAKFXH8dphWbdozccI4t9VPVFq2\nbstvvy9xSe/3yYf0+aQfG7fu4ON+/fn0E+tnW5O+G4+fX142bdvJkuV/8nGfXty5c8cl/4xpP1C3\nfkM8PT0daQM+/5Sy5StEui+vVa7C1oA9bN6+ixw5czFy2GAAxnwzkl/mLWLw0JFMmfQdAMMGD+SD\n3h/h4XH/49S0RctwJyVPipenFwMGDWPLjr2sWLORyd+FP95On/6H1atW8lKmzI60FClSMnj4KHr+\n770oy16xbAn5CxQkadKkjrSFS/9g/dYAVm/cGmm+0NBQ+vf7iNcqV3WkTf1+IoOGj2TubwsZO2ok\nAFMmTaBJs5YkSpTIsV7Hzl0ZPXL4o1fAQwTfDaXGh79Q6s2ZlHpzJtWKZ6GkX3oARvesTPshyyjd\n40fmrD5AnxbWCcE//96gy4gVzFl9wKW8oDshlO7xI6V7/Mgb/RdEuM2UPgko6efLxn1notwOQJKE\n3vRoUIRtf5+LsKwMqRLzZv0ilHvrR4p3m4GnhwdvVMxN/pdTE3QnhJLdZ1IsVzqSJopH+pSJKZHb\nl4WbjzryL912nFqlspEwvnYyupsGTzdL7+vrCD4+Pj7kyu3HubNRX3v+848V5MtfgAIFCwGQMlWq\ncF/0YX6eM4tatSM/232YX3+eTaPGVg+Gt5cXgYGB3L17F2OswWhfffkZH3/6ebg8iRIlInOWLAT4\nb4vxdh9VufKvkCJlSpd0EeGG3fq5fu0a6X1976ffvIExhpu3bpIiRUq8vFy/JObOnkXtuvfrbeeO\nAC5evEDlKlVd1g1TuUo1R1klSpbizJnTAHh7exMYFEhgUCDe3t4cO3aUM6f/ocIrFcPlr1W7Hr/M\nnR29CoiBhx1vn/R+n/4DBiNOwx/TpE1L0eIlIjxBc/bznJ+oVSf6x9vEb8dSt36jcCdzXt7eBAUG\nEhQUiJe3N9euXmXZkkU0a9k6XN4y5SqwZvUqQkJCor3dyNy6fRcAby8PvLw8sA93DIakieIBkDRx\nfM5dvgXAqQvX2Xf8EvdMzAZpNiifkxUBJxyvI9sOwGdtyjLiZ39u3438/Xp5epAwnheeHkLC+F6c\nu3yTuyH3SBjPCxHrfYXeM/RrXYYBMze75F+/5zS1SmaLoOQnR8Q9j9hMg+cTdOrkCfbs3kWxEvfP\nNCd9N55yJYvQs1snrtrdekePHEZEeL1eTV4tW4JvRg6LsLytWzaFaxUC9OjaiQqlizFs8ABHEIzM\nvF9/5vU3mgHwWuWqnDp5kqoVy9Gl+1ssWbyQQoWL4OubwSVf4aLF2LxxQ7TeuzsNGfY1/T7+kDw5\nstD3o970/+IrALp068GhAwfIle0lyhQvxJDhX4dr/QHcuXOHEyeOkSVLVgDu3bvHJ316MXBQxHUc\nkRnTf6Bqdes30+/16kPXju0YOWwIXbr14IvP+tK3/5cueVKkSEFwcDCXL1+O2ZuOgQePtyULF+Cb\nIaPjpCy6tm7eRKEixRyvRYRGdWtSsWxJpn4/KcI8Z8+cYdGC+XTs0i1ceqeubzJy2GDe7Nye93r1\nYdjgAbzXq4/L/5eHhwfZsmdn357dMdrniHh4CFvGteTU7K78ueMU2w+eB+DNr/9g3pcNODKjEy0q\n5WH43O0PKQkSxPNiw+gWrP26WbjuUWdl8mZg5+H7l1Ai207hHGl5KY0Py7Ydj3R7Zy/fYtQvARya\n0Ynjs7pw/VYwq3ac4uA/V7h0LYjNY1uyZMsxsmdIjoeHsOuI66WbHYcvUC6/6+f6SXHXSNtYHjtj\nZ/AUka9F5B2n18tFZLLT6xEiEnWfU/jybkaSPlVEGj/e3kbs5s2btGnRhEFDRzq6vTp06sbOfYdY\nvyWAdOnT0/ejXoDVbbtl80YmTpnB0j/WsnjhfNauXuVS5oXz50mdOo3j9cQpM9i0fRdLVq5h88YN\nzJk1M9L98d++lYQJE5E3X34AvLy8mDx1Jus2+9OgUWMmjP2GHm+/xycfvk/blk1YsnihI2+aNGk5\nf+6sW+olJiZPnMCgoSP4+8hJBg0dQc/unQFYtXI5BQoW4tCx02zYuoNe777tuD4X5vKlSyRLltzx\netJ331Ktek0yvvTSI2172JCv8PL0ommzlgAULFSYP9dtYvHyVZw4cYz06X0xxtCuVTM6tW/NxQsX\nHHmfZr3dvHmTNs3vH2+BgYGMHDaIj/r1j3GZV/+7go+Pj+P10j/Wsnbzdn6ev4jJE79l44Z1Lnk+\n7v0e/QcMcgmKmTJlZtHyP1mxZiOJEiXi7Jkz5Mqdh64d29KhdXOOHD7kWDd1mrScc2O93btnKN3j\nR3K0mkzx3OnJmyUVAG81KkLDfvPJ0XoyM1buZ0iXVx5aVu42kyn/9izaDlnKsG6v8vID1y8B0qdM\nzKVr96/jRrQdERjS5RU+nORah86SJ4lPnTLZyNNuCtlaTiJxAm+aVfIDoNd3aynd40e++W0Hn7Yp\nyxfTNtG7WUlmflyb9jXyO8q4eDUQ31RJHqmu3CYORM9YGTyBjUBZABHxAFID+ZyWlwU2PawQEXkm\nHf13796lbYsiB0wPAAAV2ElEQVQ3eKNpc+rWb+hIT5suHZ6ennh4eNC2fScC/K0z0AwZX6JsuQqk\nSp2aRIkSUbV6TXbv2ulSboIECQkOvj+QKEMGa0CCj48PjZs0JyAg8jPn336ew+tNmka47PuJ39Ks\nRWv8t20habJkTJn+E+NGj3QsD759mwQJE0avEtzopx+nU69BIwAavv6Gowt55oyp1KvfEBEhe/Yc\nZMn6MocOhr9OlSBhQoKdBl9t27qZiRPGkT93Nj75qDezZ83gs74fRbjdH2dMZdmSxUyeOjNctydY\ng2mGDh5I74/6MnjgF3wxcAjtOnRiwvgxjnWCg59OvTmOt2bNqdvAOt6OHzvKyZMnqFCqKAX9snP2\nzGleLVuCC+fPP3K5nl5e4QZBZchoHW9p0qalTt367PB3Pd527gigY5uWFPTLzoJ5v/LBOz1ZvOD3\ncOsM6N+PTz77gu/Gj6FNuw70HziYIV/db70H375NwidQb9duBbN29z9UK56V1MkSUuDlNI5W6C9r\nD1E6z8NbZ2ftLtcT56+xbs9pCmd3HWcQdCeE+PGsr57ItuOTMB55s6RmxdDGHJjWgZJ+vvzSv57L\noKFKRTJz4sJ1Ll0LIiT0HvM3HnHZzzqls7HzyAUSJ/Qmm28yWn21mIYVcjqucyaI50XQHfd1gytL\nbA2em4CwIWL5gH3ADft3OPGBPMBOERkmIvtEZK+INAUQkYoisl5EFgB/ORdq/75nrIgcFJE/ANcj\n/zEZY3ire2dy5c5Dj7ffDbfs/Ln7gwIWLZhPnnzW+UDlKtX4a/8+AgMDCQkJYeP6deTOk8el7Fx+\nfhw7egSwWqthoyfv3r3L8mWLyZM3n0sesLoq5//2C683dg2eV//7j+VLF9OsZWuCgoLw8PBARLgd\nFORY58iRw+TJm98l79OS3jcDG9avBWDtmj/JnsMaAZkpU2bWrPkTgIsXLnD40EFefjn8tZ0UKVIQ\nGhrqGL38/dSZ/HX4BPsOHmPgoKE0a9GazwcMctnmyhXLGDVyOHN+mR9uQEuYWT9Op1r1WqRMmZKg\nwEA8PDzwEA/HyFFjDBfOn3d0Fz8pkR1v+fIX4PDJc+w5cJQ9B46SIeNLrN20nXTp0z9y2Tlz5nKM\n7L516xY3btxwPP9z1coIj7fdfx9xbLNew9cZPmostevVdyzfuH4t6X19yZ4jJ0FBgYiHBx4eHgQ5\njbh15/GWOllCkiWOD0CCeJ5ULpqFg/9c4b8bt0maOD45Mlq9EpWKZubgP1H/OiF5kvjEs0e6pkqa\ngDJ5M/D3Kddu+YOnrpA9g1VuZNu5HniHTE0n4Nd2Cn5tp7DtwDka91/AjsMXwpX1z8UblPTzdQTC\n1wqH308vTw96NizKyJ/9SRjPi7ALN54eHsTzsvY1Z8bk/HXi6V0+gKd3b9tnKVYOwTLGnBWREBHJ\njNXK3Ix1r8EywDVgL1AHKAwUwmqZbheRsD6QokB+Y8yDFxMaArmBvFj3O/wLmBLRPohIF6w774cb\nqfgwWzZvZM5PM8mbrwAVSlvXi/r1/5JqNWrxWd8+7N2zGxEhc5YsfD3aGo2ZPEUK3nzrHSq/UhoQ\nqlavQfUatV3Krla9FhvWr6VipSoEBwfzev1a3L17l3v3Qnm1YmXatu8EwJLFC9m1w5+P+1mDfzZt\nWEfGl14KN0o3zNBBX/K+PVK0UpVqTP5uPOVKFqZ9xy6OdbZu3kSfjz995DqIqfZtWrBh/VouX7qE\nX/bMfNzvM9q068iYcd/xYa93CQkJIX78BHwzdgIAvfv0pVuX9pQuXghjDJ8PHESq1Kldyq1UpSqb\nN23gtUpVotx+z+6d6dCpK0WLFeeDd9/mTnAw9etUB6xBQ6PGWP9fgYGBzJoxnfmLlln53n6Xxg3r\nEC9ePL6fanWd79wRQImSpSIcwOROWzZvZM6smeTNX4AKpezj7fMvHSO5I3Lh/HkqlS/FjRvXEQ8P\nJowdzeYde8ONqgWoVqMWG9atJVv2HPx78QKtmllXOEJDQni9STOqVLOuA4eNOu7QuWuU+2qMYfiQ\nr5gy/ScA2nboTJf2bQgJDWHEN2MB6yQoYYIE0QryUUmfMjGT3q+Op6fgIcKv6w6x1L7G2OOblfzU\nty73jOHqzdt0HbkSgGK50jGnX12S+ySgVqls9G1dhmJdp+OXKSVj3q7CPWPwEGH43O0cOOUacJdt\nO07H2gWYumwfofdMpNuJjG/KxIx/pyoNP53P9oPnmbf+MJvHtiQk9B67j/7L90v3OtbtVrcQM//4\ni6DgEPYev0Si+F5s/7Y1y7cf59qtYABeKZSJT394umMWYvtgH3eQhw0yeVZE5EdgIdY0MiOxgmdZ\nrOCZCogP7DXGTLHXnwH8DFwHPjPGvOZU1k1jTBIRGQXsccrzGzDLGPNLVPtSpGhxs3pD5EPzn5ag\noCDq1azMslXrIxyN+yTs2bWTcWNG8d330x6+8gM8PWLHJ2jXzh2MGzOKSVOmP7Vt9n7/HWrVqUvF\n1ypHO29oLPlMnj93ju6d2zFv0UNvtuI248eMwscnKa3bdYhRft+Grr9zfRZWjWhCo09/dwSwZyVt\n8kRM/bAmtT76NdJ1gjeP5N61f9z2Yc1XsKiZvSTqa7mPqmAmn4DHubftkxRbu23h/nXPAljdtluw\nWp6Pcr3z1kOWP5cSJkxIn08+4+xDfvriTpcvX+KTB36+8rwpXKQor7xaMdxNEp60vPnyxShwxibp\nfX1p076TyyCsJylZsuQ0b9XmqW3vSekzcR2Z0vo8fMUnLFNaH/o8ZFDSkxAHxgvF6pZnYeA34Jgx\npoqdFoDVAs0PvAJ0xbrdUkrAH2syUz/gA2NMHaeywlqejZzypMXqtu38vLQ8nzexpeX5vIktLc/n\nUWxpeT5P3N7yLFTUzHFTy7PAS7G35Rkrr3na9mJdy5z1QFoSY8wlEZmH1RLdDRigtzHmvIj4RVHm\nPKASVtA8hXUtVSmllIqWWBs8jTGhQNIH0to5PTdAL/vhvM4aYM0DaUmc8vR8EvurlFLKEttHyrpD\nrA2eSimlnj9C3BhtG5sHDCmllFKxkrY8lVJKuVUcaHhq8FRKKeVmcSB6aretUkopFU3a8lRKKeVW\nOtpWKaWUiiYdbauUUkopF9ryVEop5VZxoOGpwVMppZSbxYHoqd22SimlVDRpy1MppZTbWNOJvfhN\nTw2eSiml3Ed0tK1SSimlIqAtT6WUUm4VBxqeGjyVUkq5WRyIntptq5RSSkWTtjyVUkq5kehoW6WU\nUiq6dLStUkoppVxoy1MppZTbCHFivJAGT6WUUm4WB6KndtsqpZRS0aQtT6WUUm6lo22VUkqpaNLR\ntkoppVQsJSInRGSviOwSEX87LaWIrBSRw/bfFHa6iMhoETkiIntEpOjjbFuDp1JKKbcSNz0e0WvG\nmMLGmOL26z7AKmNMTmCV/RqgJpDTfnQBvo3p+wMNnkoppdzJnpLMHY8Yqg9Ms59PAxo4pU83li1A\nchHxjelGNHgqpZSKrVKLiL/To8sDyw2wQkQCnJalM8acs5+fB9LZzzMC/zjlPW2nxYgOGFJKKeVm\nbhsxdMmpOzYi5Y0xZ0QkLbBSRA44LzTGGBEx7toZZxo8lVJKuY3w9EbbGmPO2H8visg8oCRwQUR8\njTHn7G7Zi/bqZ4BMTtlfstNiRLttlVJKPXdEJLGI+IQ9B6oB+4AFQFt7tbbA7/bzBUAbe9RtaeCa\nU/dutGnLUymllFs9pYZnOmCeWM1cL2CWMWaZiGwH5opIR+Ak0MRefwlQCzgCBALtH2fjGjwfwa6d\nAZdSJPY6+az3IxKpgUvPeieeQ1pvMaP1FjOxud6yuLvAp9Fta4w5BhSKIP0yUDmCdAP0cNf2NXg+\nAmNMmme9D5EREf+HXFBXEdB6ixmtt5jRenvxaPBUSinlVnpvW6WUUiq6XvzYqaNtXwATn/UOPKe0\n3mJG6y1mtN5eMNryfM4ZY/RDGQNabzGj9RYzca3e4kDDU4OnUkop93nM+9I+N7TbVimllIomDZ5P\niYiE2nPO7RaRHSJSNobldBORNu7ev9hORD4Rkf32PHy7RKRUJOsVF5HRTq+9ReS4nWeXiJwXkTNO\nr+NFcz8GiMg7j/t+nrVHrc9ollkxpsf180JEvnb+/xeR5SIy2en1CBF5Lxrl3YwkfaqINH68vX12\nxE3/YjPttn16gowxhQFEpDowCHg1uoUYYya4e8diOxEpA9QBihpjgkUkNRBh0DPG+AP+TknlgUXG\nmLfssvoDN40xw5/sXsde0anPaJTpBVQEbgKbHnsnY6+NWHesGSUiHlg3P0jqtLws8O7DChERL2NM\nyJPZxVggdsc9t9CW57ORFPgPHGfri8IWiMhYEWlnPx8sIn/ZrYPhdlp/EfnAfr5GRIaIyDYROSQi\nFex0TxEZJiLb7bxd7XRfEVlntzT2iUgFe92p9uu9IvLQD/4z4Is1u0IwgDHmkjHmrIiUEJFNdmt+\nm4j4PFifQA1gaVSFi0hbO/8uERlvfykiIrXtXoLdIrLCKUsBEVkrIsdExG13LHmKIqvPEyIy1D4O\ntolIDgARySoif9rH0ioRyWynTxWRCSKyFZgLdAPeteuxgoi8YR9Xu0Vk3bN6s262CShjP8+HdS/V\nGyKSQkTiA3mAnfbnL+wz1RQcn/X1IrIA+Mu5ULGMFZGDIvIHkPbpvSUVE9ryfHoSisguIAHWl1el\nqFYWkVRAQ8DPnlYneSSrehljSopILeAzoArQEeumxyXsD/RG+8u/EbDcGDNQRDyBREBhIKMxJr+9\n3ci28yytAD4VkUPAH8AcYLP9t6kxZruIJAWCIsj7GvB5ZAWLSH6sei5rjAkRkYlAMxH5E2um+QrG\nmJMiktIpWy6s238lB/4WkQnGmNDHf5tPjUt9GmPW2suuGWMKiHVpYBRWC3UMMM0YM01EOgCjuT/B\n8EtYdRf6YKteRPYC1e0po2LjcRVt9klGiH0CURbrOMyIFVCvAXux6qww1q3jUgPbnU4eigL5jTHH\nHyi6IZAbyIt1z9a/gClP+O08MXGg4aktz6coyBhT2Bjjh9Uami4S5Zi0a8Bt4HsRaYR1I+OI/Gb/\nDQCy2s+rYc0esAvYCqQCcgLbgfb2l1wBY8wN4BiQTUTGiEgN4HpM3+CTYoy5CRQDugD/YgXNrsA5\nY8x2e53rD3aDiUhG4IoxJrK6A+tkowTgb9fXq0B2rC/D1caYk3b5V5zyLDLG3DHGXASuALH29o0R\niag+w3o7gJ+c/oa1sMoAs+znM7C6wsP8HMWJw0Zgqoh0Bjzds/exwiaswBkWPDc7vd6IVT8/GWNC\njTEXgLVYxxjAtggCJ8ArTnnOAn8+4ffwRIWNuH3cR2ymLc9nwBiz2b7OlAYIIfxJTAJ7nRARKYnV\nwmkM9CTi1mqw/TeU+/+fArxljFn+4Moi8gpQG+tLbaQxZrqIFAKqY3W7NQE6POZbdDv7C3oNsMZu\n0TxKd2kNwKUOHiDAFGNMv3CJIg2jyBPs9Ny53p8bEdRn2BROzhMHP8okwrei2EY3sQYi1QYCRKSY\nfdPu591GrEBZAKvb9h/gfawTzx+wejsiE2l9vThi/2Afd9CW5zMgIn5YZ+KXsabMySsi8e2urcr2\nOkmAZMaYJVgDEFxmD4jCcqC7iHjbZeUSa+67LMAFY8wkYDJQ1A7iHsaYX4G+WN1KsYqI5BaRnE5J\nhYG/AV8RKWGv4yPWoBVnD73eidVt2cSuB0Qkld0ltwl4za4zHui2fa5FUp9hswY1dfq72X6+CWhm\nP28JrI+k6BuAj9N2shtjthpjPsVq4WaKJN/zZhNW1+wVu6V4BasLv4y9bD3QVKzxBGmwWpXbHlLm\nOqc8vkQdgFUs8NydMT/Hwq55gtXaaWuf/f8jInOxzmCPAzvtdXyA30Ukgb3+Iw9/xwqMWYEddtfw\nv1jXqCoCvUTkLtaoyDZY12t+CBskA3wUs7f3RCUBxtgnFyFY8/F1wTrLHyMiCbGud1YJy2Bf081h\njDkQVcHGmL0i8jnwh10Hd4Fu9nXU7lj/BwKcBWo+gff2LERWn3WAFCKyB6t13dxe/y2sY6QX1rEU\n2TyIC4FfRKS+neddO0gLsArY/YTez9O2F+ta5qwH0pIYYy6JyDysQLobq/Xe2xhz3j5pjsw8rJ6l\nv4BT3D9xee4Isb/L1R3EmuJMqReLiJQHWhljuj3rfXleiMgJoLgxJrbOO6meA0WKFjd/btjqlrJS\nJvYKiK1TuWnLU72QjDEbgA3Pej+UUi8mDZ5KKQCMMVmf9T6oF0Nc6LbV4KmUUsqtdLStUkoppVxo\ny1MppZT7PAc3OHAHbXmqOE3uz3azT0R+FpFEj1GW4766IlJPRPpEsW5yEXkzBttw3Nv4UdIfWCda\nM3WIdU/bfdHdRxW3iRsfsZkGTxXXhd02MT9wB+suSw72Dbuj/TkxxiwwxgyOYpXkQLSDp1IqdtDg\nqdR964EcdovroIhMx7p5RSYRqSYim8WaZeVn+w5QiEgNETkgIjuwbryPnd5ORMbaz9OJyDyxZhfZ\nLdacl4OB7Hard5i9Xi+5PxPO505lfSLWrDkbsG4eHiUR6WyXs1tEfn2gNV1FRPzt8urY60c4C49S\nMRYHmp4aPJXCMR9lTaw7xYB1I/3xxph8WPcj7QtUMcYUxZov9D377k+TgLpYN1pPH0nxo4G1xphC\nWLc/3A/0AY7ard5eIlLN3mZJrNvlFRORV0SkGNat8QoDtbh/g/Go/GaMKWFv72+sWXbCZLW3URuY\nYL8Hxyw8dvmdReTlR9iOUhHSybCVevE53zZxPfA9kAE4aYzZYqeXxpoqaqN1pz7iYd0+zQ84bow5\nDCAiM7Fuc/egSli3Qgy7Ifs1EUnxwDrV7EfY7RmTYAVTH2Be2MwwYs0F+TD5RWQAVtdwEsLfHH+u\nMeYecFhEjtnvoRpQ0Ol6aDJ724ceYVtKxUkaPFVcF2SMKeycYAdI59kvBFhpjGn+wHrh8j0mAQYZ\nY757YBvvxKCsqUADY8xusaYaq+i07MH7cRoimYVHRLLGYNtK6WhbpRQAW4ByIpIDQKwZanIBB4Cs\nIpLdXq95JPlXAd3tvJ4ikowHZiDBah12cLqWmlFE0mLNttFARBKKiA9WF/HD+ADnxJpVp+UDy94Q\nEQ97n7MBB4lkFp5H2I5SEYoDlzy15anUwxhj/rVbcD+JSHw7ua8x5pCIdAEWi0ggVrevTwRF/A+Y\nKCIdseb/7G7P6brR/inIUvu6Zx5gs93yvYl1Y/sdIjIHa4aOi1gTmj9MP6xJ0P+1/zrv0yms6bGS\nYs0ec1tEIpuFRykVCZ1VRSmllNsULVbcbNjyKOd4D5c4nofOqqKUUipuiO0jZd1Br3kqpZRS0aQt\nT6WUUm4jxI3RtnrNUymllNuIyDIgtZuKu2SMqeGmstxKg6dSSikVTXrNUymllIomDZ5KKaVUNGnw\nVEoppaJJg6dSSikVTRo8lVJKqWj6PylcxNzx3R0nAAAAAElFTkSuQmCC\n",
            "text/plain": [
              "<Figure size 504x504 with 2 Axes>"
            ]
          },
          "metadata": {
            "tags": []
          }
        },
        {
          "output_type": "stream",
          "text": [
            "              precision    recall  f1-score   support\n",
            "\n",
            "           0       0.83      0.83      0.83      4500\n",
            "           1       0.84      0.84      0.84      4500\n",
            "           2       0.88      0.93      0.91      4500\n",
            "           3       0.89      0.85      0.87      4500\n",
            "\n",
            "    accuracy                           0.86     18000\n",
            "   macro avg       0.86      0.86      0.86     18000\n",
            "weighted avg       0.86      0.86      0.86     18000\n",
            "\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "yeiD1T_QZpdk",
        "colab_type": "text"
      },
      "source": [
        "# Inference"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "z7G7vuSTZHkQ",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "import collections"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "Xeu952p4Zweb",
        "colab_type": "text"
      },
      "source": [
        "### Components"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "1GiJpWeZZwkp",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "def get_probability_distributions(probabilities, classes):\n",
        "    \"\"\"Produce probability distributions with labels.\"\"\"\n",
        "    probability_distributions = []\n",
        "    for i, y_prob in enumerate(probabilities):\n",
        "        probability_distribution = {}\n",
        "        for j, prob in enumerate(y_prob):\n",
        "            probability_distribution[classes[j]] = np.float64(prob)\n",
        "        probability_distribution = collections.OrderedDict(\n",
        "            sorted(probability_distribution.items(), key=lambda kv: kv[1], reverse=True))\n",
        "        probability_distributions.append(probability_distribution)\n",
        "    return probability_distributions"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "6EZdo-fKZwo6",
        "colab_type": "text"
      },
      "source": [
        "### Operations"
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "CLP2Vzp3Zwth",
        "colab_type": "code",
        "outputId": "f582883f-36fc-41e7-a5a7-d094f0294bda",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 85
        }
      },
      "source": [
        "# Inputs\n",
        "texts = [\"This weekend the greatest tennis players will fight for the championship.\"]\n",
        "num_samples = len(texts)\n",
        "X_infer = np.array(X_tokenizer.texts_to_sequences(texts))\n",
        "print (f\"{texts[0]} \\n\\t→ {untokenize(X_infer[0], X_tokenizer)} \\n\\t→ {X_infer[0]}\")\n",
        "print (f\"len(X_infer[0]): {len(X_infer[0])} characters\")\n",
        "y_filler = np.array([0]*num_samples)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "This weekend the greatest tennis players will fight for the championship. \n",
            "\t→ this weekend the greatest tennis players will fight for the championship \n",
            "\t→ [ 271 2033   10 5507  894  348   64  237    5   10 1615]\n",
            "len(X_infer[0]): 11 characters\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "q1gFlI5MZ143",
        "colab_type": "code",
        "colab": {}
      },
      "source": [
        "# Inference data generator\n",
        "inference_generator = DataGenerator(X=X_infer,\n",
        "                                    y=y_filler,\n",
        "                                    batch_size=BATCH_SIZE,\n",
        "                                    shuffle=False)"
      ],
      "execution_count": 0,
      "outputs": []
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "UFE4sp_7aHTq",
        "colab_type": "code",
        "outputId": "8c403d7b-7359-4f81-898b-8d82fb3a6ea7",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 34
        }
      },
      "source": [
        "# Predict\n",
        "probabilities = model.predict_generator(generator=inference_generator,\n",
        "                                        verbose=1)"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "\r1/1 [==============================] - 0s 80ms/step\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "code",
      "metadata": {
        "id": "bGi_NvbBaMap",
        "colab_type": "code",
        "outputId": "84ff5775-009a-45a3-81ea-f8aa9dc1bda4",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 238
        }
      },
      "source": [
        "# Results\n",
        "probability_distributions = get_probability_distributions(probabilities=probabilities,\n",
        "                                                          classes=y_tokenizer.classes_)\n",
        "results = []\n",
        "for index in range(num_samples):\n",
        "    results.append({\n",
        "        'raw_input': texts[index],\n",
        "        'preprocessed_input': untokenize(indices=X_infer[index], tokenizer=X_tokenizer),\n",
        "        'tokenized_input': str(X_infer[index]),\n",
        "        'probabilities': probability_distributions[index]\n",
        "                   })\n",
        "print (json.dumps(results, indent=4))"
      ],
      "execution_count": 0,
      "outputs": [
        {
          "output_type": "stream",
          "text": [
            "[\n",
            "    {\n",
            "        \"raw_input\": \"This weekend the greatest tennis players will fight for the championship.\",\n",
            "        \"preprocessed_input\": \"this weekend the greatest tennis players will fight for the championship\",\n",
            "        \"tokenized_input\": \"[ 271 2033   10 5507  894  348   64  237    5   10 1615]\",\n",
            "        \"probabilities\": {\n",
            "            \"Sports\": 0.9960770010948181,\n",
            "            \"World\": 0.0039189341478049755,\n",
            "            \"Sci/Tech\": 2.893947112170281e-06,\n",
            "            \"Business\": 1.1980721410509432e-06\n",
            "        }\n",
            "    }\n",
            "]\n"
          ],
          "name": "stdout"
        }
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "4xknbwa-bW3U",
        "colab_type": "text"
      },
      "source": [
        "<img height=\"45\" src=\"http://bestanimations.com/HomeOffice/Lights/Bulbs/animated-light-bulb-gif-29.gif\" align=\"left\" vspace=\"5px\" hspace=\"10px\">\n",
        "\n",
        "We will learn how to get a little bit of interpretabiltiy with RNNs in the next lesson on attentional interfaces."
      ]
    },
    {
      "cell_type": "markdown",
      "metadata": {
        "id": "2I4XMUfsILku",
        "colab_type": "text"
      },
      "source": [
        "---\n",
        "<div align=\"center\">\n",
        "\n",
        "Subscribe to our <a href=\"https://practicalai.me/#newsletter\">newsletter</a> and follow us on social media to get the latest updates!\n",
        "\n",
        "<a class=\"ai-header-badge\" target=\"_blank\" href=\"https://github.com/practicalAI/practicalAI\">\n",
        "              <img src=\"https://img.shields.io/github/stars/practicalAI/practicalAI.svg?style=social&label=Star\"></a>&nbsp;\n",
        "            <a class=\"ai-header-badge\" target=\"_blank\" href=\"https://www.linkedin.com/company/practicalai-me\">\n",
        "              <img src=\"https://img.shields.io/badge/style--5eba00.svg?label=LinkedIn&logo=linkedin&style=social\"></a>&nbsp;\n",
        "            <a class=\"ai-header-badge\" target=\"_blank\" href=\"https://twitter.com/practicalAIme\">\n",
        "              <img src=\"https://img.shields.io/twitter/follow/practicalAIme.svg?label=Follow&style=social\">\n",
        "            </a>\n",
        "              </div>\n",
        "\n",
        "</div>"
      ]
    }
  ]
}